Road side vehicle occupancy detection system

ABSTRACT

A system for detecting occupancy of a vehicle travelling in a direction of travel along a road. The system includes a roadside imaging device positioned on a roadside, and a first roadside light emitter, and a roadside vehicle detector. A processor is configured to receive a signal from the roadside vehicle detector, command the first roadside light emitter to emit light according to a first pattern for a first duration, command the roadside imaging device to capture images of the side of the vehicle, and compute a vehicle occupancy, in each of the captured images by determining one or more regions of interest in each of the captured images, and determining a number of visible occupants in the one or more regions of interest. The processor determines a most likely number of occupants based on each determined vehicle occupancy.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 17/465,681 filed Sep. 2, 2021, the entire contentsof which is hereby incorporated by reference.

FIELD

The improvements generally relate to the field of vehicle occupancydetection systems, and more specifically to automated vehicle detectionoccupancy systems and methods.

INTRODUCTION

Determining vehicle occupancy typically includes the use of a physicalhuman presence, such as police or other policy enforcement personnel, toaccurately determine a vehicle occupancy.

Automated vehicle occupancy detection suffers from a lack of accuracyand potential latency issues. Automated vehicle occupancy detection alsosuffers from a lack of accuracy associated with various road conditionswhich are likely to occur during operation. Traditional automatedvehicle occupancy systems can also be expensive to implement, experienceimpaired functioning when moved, or be unreliable and difficult torepair or re-calibrate.

Automated vehicle occupancy detection systems which are more accurate,faster, more reliable, easier to move or install, more robust, orrequire less calibration are desirable.

SUMMARY

In accordance with one aspect, there is provided a system for detectingoccupancy of a vehicle travelling in an expected direction of travelalong a road. The system involves a first roadside imaging devicepositioned on a roadside, having a first field of view of the road, thefirst field of view incident on a side of the vehicle when the vehicleis on the road within the first field of view; a first roadside lightemitter emitting light towards vehicles in the first field of view; aroadside vehicle detector; a processor, in communication with a memory,configured to: receive a signal from the roadside vehicle detectorindicating that the vehicle is within the first field of view orproximate, relative to the expected direction of vehicle travel, to thefirst field of view; command the first roadside light emitter to emitlight according to a first pattern for a first duration; command thefirst roadside imaging device to capture one or more images of the sideof the vehicle according to a second pattern associated with the firstpattern, during a second duration associated with the first duration;receive the captured images of the side of the vehicle from the firstroadside imaging device; compute a vehicle occupancy of the vehicle by,in each of the captured images: determining one or more regions ofinterest of the vehicle in each of the captured images; determining thevehicle occupancy as a number of visible occupants in the one or moreregions of interest; and determining a most likely number of occupantsbased on each determined vehicle occupancy; and transmit the vehicleoccupancy to a monitoring system.

In some embodiments, the first roadside imaging device is positioned toextract data for different perspectives of occupants as the vehicletravels horizontally across the field of view; and each of the imagescaptured by the first roadside imaging device include differentperspectives of the side of the vehicle.

In some embodiments, the processor is configured to compute a yaw anglerelative to a horizontal axis perpendicular to the expected direction ofvehicle travel, wherein the images captured by the first roadsideimaging device include the different perspectives of the side of thevehicle based on the first yaw angle.

In some embodiments, the processor, to compute the vehicle occupancy ofthe vehicle, is configured to: discard uninteresting regions of theplurality of captured images to generate subsets of the plurality ofcaptured images; and determine the number of visible occupants based ondetermining one or more regions of interest of the vehicle in therespective subset of the plurality of captures images.

In some embodiments, the first roadside imaging device, the firstroadside light emitter, and the vehicle detector are attached to amobile roadside structure.

In some embodiments, the system has a second roadside imaging device,above the first roadside imaging device, the second roadside imagingdevice having a second field of view of a second lane of the road, thesecond lane being further from the first roadside imaging device than afirst lane of the road, the second field of view incident on a side of afurther vehicle when the further vehicle is in the second lane withinthe second field of view; a second roadside light emitter adjacent tothe road and emitting light towards vehicles in the second field ofview; wherein the processor is further configured to: receive anothersignal from the vehicle detector indicating that the further vehicle iswithin or proximate, relative to the expected direction of vehicletravel, to the second field of view; command the second roadside lightemitter to emit light according to a third pattern for a third duration;command the second roadside imaging device to capture additional imagesof the side of the further vehicle according to a fourth patternassociated with the third pattern, during a fourth duration associatedwith the third duration; receive the additional captured images of theside of the further vehicle from the second roadside imaging device;compute a vehicle occupancy of the further vehicle by, in each of theadditional captured images by: determining one or more regions ofinterest of the further vehicle in each of the additional capturedimages; determining the vehicle occupancy of the further vehicle as anumber of visible occupants of the further vehicle in the one or moreregions of interest of the further vehicle; and determining a mostlikely number of occupants of the further vehicle based on eachdetermined vehicle occupancy of the further vehicle; and transmit thevehicle occupancy of the further vehicle to the monitoring system.

In some embodiments, the first field of view and the second field ofview overlap, and the processor is further configured to: determine theone or more regions of interest of the vehicle in the one or moreadditional captured images; determine a further number of visibleoccupants of the vehicle in the one or more additional captured imagesin the one or more regions of interest; and determine the most likelynumber of occupants of the vehicle based on each determined vehicleoccupancy and each determined further number of visible occupants.

In some embodiments, the processor is further configured to: monitor,over time, a plurality of signals from the roadside vehicle detector todetermine an expected vehicle speed of the vehicle; and adjust one ormore parameters of the first roadside imaging device or the first lightemitter into a determined optimal configuration for capturing vehiclestravelling the expected vehicle speed.

In some embodiments, the processor is further configured to: monitor,over time, a plurality of signals from the roadside vehicle detector todetermine an expected vehicle speed of the vehicle; and determine thefirst pattern and the first time window based on the expected vehiclespeed.

In some embodiments, the system has a sensor for detecting ambientconditions; wherein the processor is further configured to: receiveambient condition information from the sensor; determine an optimalconfiguration for the imaging device based on the received ambientcondition; and transmit a further command signal to the imaging devicecapture images according to the optimal configuration.

In some embodiments, the light emitter is an LED emitting infrared ornear infrared light, the first pattern is 120 pulses per second, and theregions of interest are a rear side window and a front side window.

In accordance with another aspect, there is provided a method fordetecting occupancy of a vehicle travelling in an expected direction oftravel along a road. The method involves receiving a signal indicatingthat the vehicle is within or proximate, relative to the expecteddirection of vehicle travel, to a first field of view of a firstroadside imaging device; commanding a first roadside light emitter toemit light according to a first pattern for a first duration; commandingthe first roadside imaging device to capture images of a side of thevehicle according to a second pattern associated with the first pattern,during a second duration associated with the first duration; receivingthe captured images of the side of the vehicle from the first roadsideimaging device; computing a vehicle occupancy of the vehicle by, in eachof the captured images: determining one or more regions of interest ofthe side of the vehicle in each of the captured images; determining thevehicle occupancy in the one or more regions of interest; anddetermining a most likely number of occupants based on each determinedvehicle occupancy; and transmitting the most likely number of occupantsto a monitoring system.

In some embodiments, the method involves discarding uninterestingregions of the plurality of captured images to generate subsets of theplurality of captured images; and determining the number of visibleoccupants based on determining one or more regions of interest of thevehicle in the respective subset of the plurality of captures images.

In some embodiments, the one or more regions of interest include atleast one of a rear side window and a front side window.

In some embodiments, each of the captured images includes the side ofthe vehicle at different perspectives based on a yaw angle whichencourages image variation.

In some embodiments, the method involves commanding a second roadsideimaging device to capture additional images of the side of the vehiclefrom a second field of view according to a fourth pattern associatedwith the first pattern, for a fourth duration associated with the firstduration; receive the additional captured images of the side of thevehicle from the second roadside imaging device; wherein computing thevehicle occupancy of the vehicle further comprises, for each of theadditional captured images: determining one or more additional regionsof interest of the vehicle; determining the vehicle occupancy of thevehicle in the additional one or more regions of interest of thevehicle; and determining the most likely number of occupants of thevehicle based on the each of the number of visible occupants and thefurther number of visible occupants; and transmitting the vehicleoccupancy of the further vehicle to the monitoring system.

In some embodiments, the method involves receiving a signal indicatingthat a further vehicle is within or proximate, relative to the expecteddirection of vehicle travel, to the second field of view; commanding asecond roadside light emitter to emit light according to a third patternfor a third duration; commanding the second roadside imaging device tocapture additional images of a side of the further vehicle according toa fourth pattern associated with the third pattern, during a fourthduration associated with the third duration; receiving the additionalcaptured images of the side of the vehicle from the first roadsideimaging device; computing a vehicle occupancy of the further vehicle by,in each of the additional captured images: determining one or morefurther regions of interest of a side of the further vehicle in each ofthe additional captured images; determining the further vehicleoccupancy as a number of visible occupants in the one or more furtherregions of interest; and determining a most likely number of occupantsof the further vehicle based on each determined further vehicleoccupancy; and transmitting the most likely number of occupants of thefurther vehicle to the monitoring system.

In some embodiments, the method involves computing a correctionparameter and providing visual guidance using augmented reality avatarson a display device.

In some embodiments, the method involves monitoring, over time, aplurality of signals from the roadside vehicle detector to determine anexpected vehicle speed of the vehicle; and adjusting one or moreparameters of the first roadside imaging device or the first lightemitter into a determined adjusted configuration for capturing vehiclestravelling the expected vehicle speed.

In some embodiments, the method involves monitoring, over time, aplurality of signals from the roadside vehicle detector to determine anexpected vehicle speed of the vehicle; and determining the first patternand the first time window based on the expected vehicle speed.

Many further features and combinations thereof concerning embodimentsdescribed herein will appear to those skilled in the art following areading of the instant disclosure.

DESCRIPTION OF THE FIGURES

In the figures,

FIG. 1 is a network diagram of a system for vehicle occupancy detection,in accordance with example embodiments;

FIG. 2 is an example schematic diagram of a system for vehicle occupancydetection, in accordance with example embodiments;

FIG. 3 is another example schematic diagram of a system for vehicleoccupancy detection, in accordance with example embodiments;

FIG. 4 is a flowchart of an example method for configuring a system forvehicle occupancy detection, in accordance with example embodiments;

FIG. 5A, is a further example schematic diagram of a system for vehicleoccupancy detection, in accordance with example embodiments;

FIG. 5B is a perspective view of an example system for vehicle occupancydetection, in accordance with example embodiments;

FIG. 5C is a perspective view of the system of FIG. 5B including anotherimaging device, in accordance with example embodiments;

FIG. 5D shows a photograph of the system 514 of FIG. 5B, in accordancewith example embodiments;

FIG. 6A is a top view of an example system for vehicle occupancydetection with a vehicle in a first position, in accordance with exampleembodiments;

FIG. 6B is a top view of an example system for vehicle occupancydetection with a vehicle in a second position, in accordance withexample embodiments

FIG. 6C is a top view of an example system for vehicle occupancydetection with a vehicle in a third position, in accordance with exampleembodiments

FIG. 6D is a rear view of the example system for vehicle occupancydetection of FIG. 6A, in accordance with example embodiments;

FIG. 7 is a perspective view of a further example system for vehicleoccupancy detection, in accordance with example embodiments;

FIG. 8 is a flowchart of an example method for vehicle occupancydetection, in accordance with example embodiments;

FIG. 9 is a flowchart of an example method to complete step 812 of FIG.8 for detecting occupants in images, in accordance with exampleembodiments;

FIGS. 10A to 10G are each an image of a vehicle with various regions ofinterest shown, in accordance with example embodiments;

FIG. 11 is an example report interface for viewing vehicle occupancy,according to example embodiments;

FIG. 12 is an architecture diagram of the system of FIG. 1 , accordingto example embodiments;

FIG. 13 is an example reviewing interface for vehicle occupancydetection validation, according to example embodiments; and

FIG. 14 is an example schematic diagram of a computing device, inaccordance with an embodiment.

FIG. 15A is an example diagram of vehicle weaving.

FIG. 15B is an example schematic diagram of vehicle detector sensor fordetecting vehicle weaving in accordance with an embodiment.

FIG. 16 is an example schematic diagram of a cloud server and road sideunits in accordance with an embodiment.

FIG. 17 is an example schematic diagram of a cloud server and road sideunits in accordance with an embodiment.

DETAILED DESCRIPTION

Embodiments described herein provide computer vision based VehicleOccupancy Detection (VOD) systems and methods. The described systems caninclude a road-side unit (iRSU) with an imaging device that capturessuccessive images of vehicles moving along in a lane on a highway orroad. The road-side unit is configured to capture a plurality of imagesof one or more vehicles travelling along a lane of a road. The road sideunit can capture images from a fixed perspective. The successive imagesare, for each vehicle, analyzed to determine a likely vehicle occupancy.The system may achieve high accuracy, in some instances above 95%performance, despite processing images received from a fixed perspectiveimaging device on the side of the road. As a result of capturingmultiple images from the fixed perspective, and further as a result ofthe images being captured from a roadside position, the images betweeninstallations may allow for more robust training, and portable occupancydetection approaches, which are adaptable to a variety of operatingenvironments. The use of the multiple images being captured from a fixedroadside position also allows the system to generate a robust estimationof the vehicle occupancy without the need for expensive or overheadsystems that are difficult to install. The roadside system may requirefewer parts, have lower maintenance costs, and be easier to deploy.

Furthermore, and as a result of the unit being roadside, the system mayoperate with enhanced privacy without the need to transmit dataremotely. In some embodiments, the described system may permit rapidset-up or installation, e.g., in less than 1 hour per site, without theneed for further post-installation site specific training or tuning. Thedescribed system may further be a stationary system, or the system canbe a mobile system capable of being reinstalled or configured forvarious sites.

In example embodiments, the system has a roadside unit with a lightdetection and ranging (LIDAR) unit. In example embodiments, the roadsideunit can determine whether a vehicle is in an upstream portion of thelane, and an infrared light emitter to illuminate vehicle occupantsthrough a region of interest of the vehicle (e.g., a windshield). Theinfrared light emitter may overcome, at least to some degree, windowtint and sun-related over or under exposure. For example, a difficultlighting condition tends to arise in the daytime due to interferencefrom the sun, white washing images. The system is capable of adjustingthe imaging device parameters (e.g., number of pictures taken for eachvehicle, camera exposure time, camera frame rate) and the infrared lightemitter parameters (e.g., illumination intensity) based on measuredambient conditions (e.g., measured with ambient environmental sensorsattached to the system, or retrieved from a network) in order tomaximize the quality of the image acquisition, which in turn leads tohigher overall accuracy.

The system further comprises an infrared camera to capture images of thevehicle (and vehicle occupants), the captured images capturing at leastsome of the light emitted by the infrared light emitter and reflected bythe vehicle or vehicle occupants. Optionally, the system can include asecond imaging device (and corresponding infrared illumination source)to capture vehicle occupancy in a further lane of the road or highway(e.g., to detect occupancy in a second lane of a highway). Optionally,the system may include an imaging device for capturing vehicle licenseplates (LPR).

In example embodiments, a processor running VOD software determines afront and a rear occupancy of the vehicle from the images, and securelytransfers images and metadata to a tolling system. In exampleembodiments, the processor may determine or identify violating vehicles'license plates, and further transmit the identified license plates tothe tolling system. The system can have rules for defining parametersfor violations, for example.

The proposed system may be able to operate, unattended, under allweather conditions. Operation of the system may be transparent to roadusers, as the roadside unit does not visually distract the driver, orimpede a line of sight needed by the driver to navigate traffic.

To maintain privacy, vehicle or occupant data can be filtered andremoved so that it is not retained by the system or transmitted to acloud-based back-end service: only data related to suspected violationscan be securely uploaded to the tolling systems before local deletion.License plate recognition is implemented by software of the system, withinstallation, maintenance and consistent configuration at every site.

The system may operate with a single imaging device adjacent to theroad, as compared to a plurality of imaging devices proximate to theroad, or overhead imaging devices and so forth.

The single imaging device system requires taking successive high-speedimages as the vehicle passes the field of view of the imaging device.Subsequently, the images are analyzed to determine both front and rearoccupancy. This solution has the advantage of simplifying the system,introducing less redundancy, which in turn improves accuracy andsignificantly reduces the overall cost of the system.

The described system may be relatively simple to adjust or implement ina variety of environments as a result of the relatively fixed geometryassociated with the single configuration. In training the system priorto installation, a plurality of images are captured with the singleimaging device in a calibration unit from multiple sites, each siteexperiencing a variety of weather phenomena (e.g., rain, snow, etc.),and capturing a variety of vehicle occupant behaviors (e.g., turnedhead, wearing face coverings, etc.). Training the system can compriselabelling the training images, and subsequently training a machinelearning data model with the captured training images. The machinelearning model learns to detect vehicle occupancy based on the trainingimages. As the training images are captured in an environment which isreplicated during installation, namely the single image camera geometry,and as the single camera implementation requires far fewer variables toadjust relative to a multi-camera implementation (e.g., only one pitch,and yaw of a single camera to adjust, which does not require complicatedtuning to permit inter-camera image integration), the system mayreplicate the trained machine learning model on each system operating ina new or different site, without the need for extensive retraining ortuning of the machine learning model. Alternatively stated, the trainedmachine learning model may be robust and portable to various sites, inpart as a result of the consistency of the sites (e.g., all roads havelanes), and in part as a result of the complexity-reducing configurationof the system (e.g., the system uses a single imaging device whichcaptures multiple images).

In an illustrative example, the training process may determinerecommended configurations of the system, including a height that isrelative to the road, relative distance (horizontal and vertical)between the system components (e.g., any of the imaging device,illumination device, and LI DAR), and a pitch, yaw and roll of thesystem components. In example embodiments, the recommended configurationmay include an error parameter, an amount to which the describedparameters may be incorrect while maintaining an adequate performancelevel. For example, the system may permit geometry variations of up to15 cm for the height and x-y coordinates of the system components, andup to 10 degrees of variation in the pitch, yaw and roll of the systemcomponents.

Continuing the example, once on site, the system receives differentmeasurements: (1) a distance to the target lane (e.g., a distance fromthe intended mounting location of the unit to the centerline of thelane), (2) a width of the target lane, and (3) a height reflective ofthe difference in height of the ground between the unit mountinglocation and the target lane (e.g., in case there is a slope from thetarget lane to the mounting location).

During installation, three measurements are entered into a userinterface to the system, and a processor of the system computes arequired geometry of each component of the system to implement thetrained machine learning model with an acceptable degree of accuracy.Because the geometry for any given site is typically a small variationrelative to the training geometries (e.g., most road lane widths andheights are relatively fixed). As a final step, the imaging deviceparameters (e.g., a camera zoom, gain, etc.) can be adjusted to therecommended level provided by the automated program and locked intoposition, and the system can then begin operation.

In a diagnostic mode, live images are displayed on a diagnostic computer(laptop) that can be securely connected to the system. If no furtheradjustment is necessary, the system can begin its operation.

The system may be able to support the capture of high-quality images inrapid succession from closely following vehicles travelling in excess of200 km/h. The system uses efficient deep neural network software toachieve accuracy and handle difficult situations such as poor weatherconditions (e.g., heavy rain, heavy snow, fog), and difficult situations(e.g., children in car seat, heavy tint, etc.).

In example embodiments, the system may be installed on either a roadsidepost or gantry-mounted. The system may also be deployed in a securemobile trailer, or other mobile unit, that may be quickly moved from onelocation to the next for rapid adhoc applications. Each component of thesystem could also be separately mounted on the mobile unit providing thesystem high installation flexibility.

In example embodiments, the system can be updated remotely by a controlsystem to enable further training and tuning of the machine learningmodel. In example embodiments including multiple systems (e.g., multipleroadside units), each system may be updated by the control system.

Reference will now be made to the figures.

FIG. 1 is a network diagram of a system 100 for vehicle occupancydetection, in accordance with example embodiments.

The system 100 includes a computing device 102, a vehicle detector(s)114, a light emitter(s) 116, imaging device(s) 118 for detecting vehicleoccupancy in a first lane on a road and, optionally, second lightemitter(s) 124, an ambient condition sensor 130, and second imagingdevice(s) 126. The second light emitter(s) 124, and the second imagingdevice(s) 126 may be used to detect vehicle occupancy in a second lane,or to detect a vehicle license plate. The system 100 may require fewerparts and lower maintenance costs as compared to systems which usemultiple imaging devices to determine vehicle occupancy in a singlelane.

Vehicle detector(s) 114 can be various devices which are capable ofdetecting the presence of a vehicle at various distances. For example,the vehicle detector(s) 114 may be a laser-based system for detectingvehicles, such as a Light Detection and Ranging system (LiDAR), whichboth emits light and detects the presence of return light in response tothe emitted light reflecting off of a vehicle. According to someembodiments, for example, the vehicle detector(s) 114 may be a radiowave based system, such as Radio Detection and Ranging (RADAR), or amechanical, micro-electromechanical system (MEMS), solid-state or hybridLiDAR unit.

Vehicle detector(s) 114 may include multiple instances of devicescapable of detecting the presence of the vehicle. For example, thevehicle detector(s) 114 may include two separate LiDAR units, whichallows for greater robustness of the system as there is more returnlight to be analyzed.

Vehicle detector(s) 114 may be configured to detect one or more vehiclesat various distances. For example, a LiDAR vehicle detector(s) 114 canbe configured to ignore any readings representative of objects more than10 m away. In some embodiments, for example, where the vehicledetector(s) 114 include multiple devices, each of the multiple devicescan be configured to detect vehicles at different distances.Alternatively, the multiple devices may be used redundantly to detectvehicles a single distance away from the vehicle detector(s) 114.

The vehicle detector(s) 114 may be modified or augmented to withstandambient conditions. For example, the vehicle detector(s) 114 may beweatherproofed with various materials, such as plastic covering orcoatings, to protect against rain, snow, dust, insects and so forth.

The light emitter(s) 116 can include various devices capable of emittingspecific ranges of light at specific frequencies (i.e., patterns) forspecific durations. For example, the light emitter(s) 116 can be astrobe light configured to emit a white light at a specific frequencybased on strobe light cool down limitations.

In an illustrative example, light emitter(s) 116 includes an infraredlight-emitting diode (LED) configured to emit infrared light in therange of 750 nm to 1300 nm, or as another example, a range of 850 nm+/−10 nm. Advantageously, infrared light emitter(s) 116 may be able toilluminate the inside of a vehicle, overcoming window tint and sunlightexposure.

Continuing the example, the infrared LED light emitter(s) 116 may beable to overcome cool down limitations of strobe lights, and burstinfrared light at a rate of 120 pulses a second. Various frequencies ofpulsing are contemplated. In some embodiments, for example, the infraredLED light emitter(s) 116 may be configured to, or remotely controlled todynamically change the pattern of light emission. For example, theinfrared LED light emitter(s) 116 may pulse at different frequencies inresponse to being controlled, based on the detected speed of a detectedvehicle speed (e.g., light may be emitted faster in response to a highervehicle speed being detected).

Varying types and amounts of light emitter(s) 116 may be used in system100. For example, in the shown embodiment in FIG. 2 , the lightemitter(s) 116 includes the strobe light emitter(s) 116-1. In theembodiment shown in FIG. 3 , the light emitter(s) 116 includes the firstand second occupant light emitter(s) 116-1.

Imaging device(s) 118 (hereinafter referred to as passenger imagingdevices) can include any type of imaging device capable of capturing thelight emitted by the light emitter(s) 116. For example, where the lightemitter(s) 116 is an infrared light emitter, the imaging device(s) 118is an infrared imaging device. In some embodiments, for example, theimaging device(s) 118 may be adapted for the specific frequency of lightbeing emitted by light emitter(s) 116. The imaging device(s) 118 may bea high speed imaging device, capable of taking successive images withina short period of time.

The imaging device(s) 118 may be configured (as described herein) tocapture one or more images of one or more vehicles within their field ofview, when the vehicles are detected by the vehicle detector(s) 114. Theimaging device(s) 118 may be configured to capture multiple images(i.e., a plurality of images) upon receiving a control command to startcapturing images. For example, the imaging device(s) 118 may, inresponse to receiving a control command, capture 5 successive images at90 frames per second (FPS).

The imaging device(s) 118 are positioned relative to the road to captureimages of at least the side of the vehicle. In example embodiments, theimaging device(s) 118 are positioned relative to the road to captureimages of various combinations of the front of the vehicle, the side ofthe vehicle, the rear of the vehicle, and so forth. For example, theimaging device(s) 118 may capture one image of the front and side of thevehicle, three images of the side of the vehicle, and one image of therear and side of the vehicle.

Optionally, the system 100 may include the second light emitter(s) 124,and the second imaging device(s) 126, similar to light emitter(s) 116and imaging device(s) 118. The second light emitter(s) 124, and thesecond imaging device(s) 126 may be positioned relative to the roadsimilar to the light emitter(s) 116 and the imaging device(s) 118 butdirected to capture one or more images of the rear of the vehicle toinclude a license plate of the detected vehicle. In example embodiments,the second light emitter(s) 124, and the second imaging device(s) 126are similar to light emitter(s) 116 and imaging device(s) 118,positioned relative to the road to capture images of the side of afurther vehicle travelling in a second lane.

Optionally, the system 100 may include the ambient condition sensor 130,which detects ambient conditions such as sunlight intensity,temperature, moisture, humidity, rainfall, snowfall, and so on. Inexample embodiments, the ambient condition sensor 130 includes a varietyof sensors for various ambient conditions.

Referring now to computing device 102, the computing device 102 may beconfigured to communicate command signals to the vehicle detector(s)114, the light emitter(s) 116, the imaging device(s) 118, the secondlight emitter(s) 124, second imaging device(s) 126, and the ambientcondition sensor 130, and to receive captured images from the imagingdevice(s) 118 and second imaging device(s) 126 and detected conditionsfrom the ambient condition sensor 130. The computing device 102 may beconfigured to operate with the Linux operating system.

In example embodiments, the computing device 102 is in a housing (notshown), and is also used to transmit power to one or more of the vehicledetector(s) 114, the light emitter(s) 116, the imaging device(s) 118 thesecond light emitter(s) 124, the imaging device(s) 118 and the secondimaging device(s) 126. For example, the computing device 102 may powerthe imaging device(s) 118.

The computing device 102 may include various combinations of a vehicledetector controller 104, a light emitter controller 106, an imagingdevice controller 108, an occupant detector 110, a database(s) 112, and,optionally, a second imaging device controller 122.

The vehicle detector controller 104 can be configured to control thevehicle detector(s) 114 through a series of command signals. The commandsignals are interpretable by the vehicle detector(s) 114, and caninclude instructions to control various operating features of thevehicle detector(s) 114. For example, the command signals may adjust athreshold indicative of detection of a vehicle (e.g., certainty ratemust be over 90%) used by the vehicle detector(s) 114 to determinewhether a vehicle is detected. The command signals may control thedistance to which the vehicle detector(s) 114 operate (e.g., vehiclesthat are more than 10 m away from the vehicle detector(s) 114 will beignored), the frequency and timing of operation of the vehicledetector(s) 114 (e.g., pulse light at a first frequency to detect avehicle), and so forth.

In a non-limiting example embodiment, the vehicle detector controller104 may transmit configuration characteristics to the vehicledetector(s) 114, allowing an operator to change the operation of thevehicle detector(s) 114 through the use of the computing device 102. Forexample, where the vehicle detector(s) 114 is mounted at a first height,the vehicle detector controller 104 may transmit a calibration parameterto adjust detection of vehicles by the vehicle detector(s) 114 based onthe first height. Continuing the example, the vehicle detector(s) 114may be configured to expect vehicles at the detection distance to benear or close to the top of a field of view of the vehicle detector(s)114.

In some embodiments, for example, the vehicle detector controller 104may transmit command signal to the vehicle detector(s) 114 to detect thespeed of a vehicle. The vehicle detector(s) 114 may, in response,provide the vehicle detector controller 104 with two detections of thesame car at different instances in time, allowing for the speed to beinterpolated. In example embodiments, the vehicle detector(s) 114 iscontinuously monitoring detected vehicles in its field of view, anddirectly computes the speed of the detected vehicles and relays the sameto the vehicle detector controller 104.

The light emitter controller 106 is configured to control the lightemitter(s) 116 through a series of command signals. The command signalsare interpretable by the light emitter(s) 116, and can include commandsignals to control the type of light emitted (e.g., emitted light shouldbe 800 nm), command signals to control the power used by the lightemitter(s) 116 (e.g., increase or decrease the intensity of the emittedlight), the frequency and timing of operation of the light emitter(s)116 (e.g., pulse light at a first frequency, for a first duration,etc.), and so on.

In non-limiting example embodiments, the light emitter controller 106may transmit configuration characteristics to the light emitter(s) 116,allowing an operator to change the operation of the light emitter(s) 116through the use of the computing device 102. For example, where thelight emitter(s) 116 is capable of adjusting a field of view (e.g., suchas being able to rotate around a first axis), the light emittercontroller 106 may transmit a command signal to adjust the field of view(e.g., a command signal to swivel the light emitter(s) 116) of the lightemitter(s) 116.

The imaging device controller 108 is configured to control how theimaging device(s) 118 capture one or more images through one or morecommand signals. The command signals are interpretable by the imagingdevice(s) 118, and can include command signals to control a frequency ofcapturing images (e.g., capture two images per second) or the timing ofoperation of the imaging device(s) 118 (e.g., capture images at thistime, where the detected vehicle is expected to be within the imagingdevice(s) 118 field of view, for this duration), alter or adjust theoperating focal distance (e.g., focus is directed towards the areabetween lanes within a road), the exposure settings (e.g., the aperture,ISO and shutter speed settings), and so forth.

Each of the vehicle detector controller 104, the light emittercontroller 106, and the imaging device controller 108 may be configuredto transmit command signals to the respective devices dynamically (e.g.,in real time), at intervals, upon configuration, or some combination ofthe aforementioned. For example, the vehicle detector controller 104 maytransmit command signals to the vehicle detector(s) 114 upon powering onof the system 100, and subsequently transmit command signals to adjust adistance detection dynamically in response to changing ambientconditions.

The occupant detector 110 is configured to receive the plurality ofimages from the imaging device(s) 118 (and possibly second imagingdevice 126) and determine a number of occupants in the detected vehicle(alternatively referred to as a vehicle occupancy). The occupantdetector 110 may be a machine learning model trained to determine thevehicle occupancy based on roadside images.

The occupant detector 110 may further output the determined vehicleoccupancy upon determination. For example, the occupant detector 110 mayoutput the determined vehicle occupancy to a tolling system, which tollsindividuals based on the number of occupants in a car. In a non-limitingembodiment, the occupant detector 110 may output the determined vehicleoccupancy and the determined lane to a vehicle tolling system, whichtolls individuals based on whether a vehicle occupancy complies with theoccupancy requirements for the specific lane.

The occupant detector 110 may also coordinate or control the vehicledetector controller 104, the light emitter controller 106, and theimaging device controller 108. For example, the occupant detector 110may make determinations as to the relative offsets between the operationof light emitter(s) 116 and the imaging device(s) 118, and relay therequired offsets to the light emitter controller 106 and the imagingdevice controller 108, respectively.

According to example embodiments, the vehicle detector controller 104,the light emitter controller 106, the imaging device controller 108 andthe second imaging device controller 122 may be located within therespective unit being controlled, and not within the computing device102. For example, the vehicle detector controller 104 may be integratedwithin the vehicle detector(s) 114 and pre-configured with operationalsettings.

Continuing the example, some or all of the vehicle detector controller104, the light emitter controller 106, the imaging device controller 108and the second imaging device controller 122 may be interconnected withone another and relay command signals between each other. For example,the vehicle detector controller 104 which is integrated within thevehicle detector(s) 114 may receive command signals from the occupantdetector 110, and relays the command signals to a light emittercontroller 106 within a light emitter(s) 116 directly.

Optionally, the computing device 102 may include a sensor health monitor(not shown), which monitors the relative health of the sensor. Forexample, the sensor health monitor may notice a decrease in performanceby the vehicle detector(s) 114 based on usage, and so forth.

In response to determining sensor deterioration, the sensor healthmonitor may be configured to provide a calibration parameter to any oneof the components of the system 100. For example, in response todetermining light emitter(s) 116 deterioration, the sensor healthmonitor may instruct the light emitter controller 106, occupant detector110, or imaging device controller 108 to adjust the operation of therespective components.

The computing device 102 and the vehicle detector(s) 114, lightemitter(s) 116, imaging device(s) 118, second light emitter(s) 124, theambient condition sensor 130, and the second imaging device(s) 126 areinterconnected (e.g., transmit or receive command signals) by way of thecommunication network 120. Communication network 120 may include apacket-switched network portion, a circuit-switched network portion, ora combination thereof. Communication network 120 may include wiredlinks, wireless links such as radio-frequency links or satellite links,or a combination thereof. Communication network 120 may include wiredaccess points and wireless access points. Portions of communicationnetwork 120 could be, for example, an IPv4, Pv6, X.25, IPX or similarnetwork. Portions of network 120 could be, for example, a GSM, GPRS, 3G,LTE or similar wireless networks. Communication network 120 may includeor be connected to the Internet. When communication network 120 is apublic network such as the public Internet, it may be secured as avirtual private network.

In embodiments where the communication network 120 includes wired links,the wired links, similar to the vehicle detector(s) 114, may beweather-proofed with coatings or covering.

The system 100 may be a mobile system for vehicle detection. Forexample, the system 100 may be capable of being disassembled and movedto another location along a road. In some embodiments, variouscomponents of the system 100 may be relocated. Alternatively, the system100 may be stationary, and fixed to a fixture.

The system 100 may be configured to receive (via query or pushmechanism) one or more updated operating parameters via thecommunication network 120. For example, the system 100 may receive newparameters for calibrating the occupant detector 110, or the imagingdevice controller 108, and so forth.

FIG. 2 is an example schematic diagram 200 of the system of FIG. 1 forvehicle occupancy detection, in accordance with example embodiments.

In the shown example embodiment, the system includes the computingdevice 102, a laser sensor vehicle detector(s) 114, a strobe lightemitter(s) 116-1, a passenger camera imaging device(s) 118, a strobelight second light emitter 124, and a plate camera second imaging device126.

The example implementation further includes a network switch 230, and apower supply 232 (shown as universal AC/DC power supply). The networkswitch 230 may be used by the computing device 102 to transmit commandsignals, and the network switch 230 may use packet switching to receivecommand signals from the computing device 102 and forward said commandsignals to the destination component.

The power supply 232 may be one or more devices of various types capableof powering the network switch 230, computing device 102, the lasersensor vehicle detector(s) 114, the strobe light emitter(s) 116-1, thepassenger camera imaging device(s) 118, the strobe light second lightemitter 124, and the plate camera second imaging device 126. In someembodiments, for example, the power supply 232 includes multiple powersupply units (not shown). For example, the power supply 232 may includevarious types of batteries. The power supply 232 may also be connectedto an AC power source, such as a power line in road infrastructure.According to example embodiments, the power supply includes 80-264 VAC,derate output power 10% <90 VAC, and 20% <85 VAC350 & 1000 W: 85-264VAC, derate output power 10% <90 VAC.

In example embodiments, the power supply 232 includes the ability toconvert received power into power as required by the components of thesystem. For example, the power supply 232 may include a universal AC/DCpower supply, capable of converting stored or received AC power into DCpower, and providing the DC power as required by the components of theimplementation 200.

FIG. 3 is another example schematic diagram 300 of the system forvehicle occupancy detection of FIG. 1 , in accordance with exampleembodiments. In FIG. 3 , the power source 302 is a power supply lineincorporated into road infrastructure, such as the power supply linewhich provides power to roadside signage.

Power supply 304, similar to power supply 232, may be configured tocovert the power received from the source 302 into a form usable by thecomponents of the system 100. For example, the power supply 304 mayprovide power to the computing device 102, which may in turn includeadditional electronics for providing power to one or more of the LiDARvehicle detector(s) 114, the light emitter(s) 116-1, the imagingdevice(s) 118, the second light emitter 124, and the second imagingdevice 126.

In the shown embodiment, the computing device 102 is used as a conduitto provide power to the light emitter(s) 116-1, and the second lightemitter 126.

According to some embodiments, for example, in the diagram 300, thecomputing device 102 sends power and command signals to the vehicledetector(s) 114, the imaging device(s) 118, and the second imagingdevice 126 which include command signals for the light emitter(s) 116-1and second light emitter 124. Upon receipt of the command signals, thevehicle detector(s) 114, the imaging device(s) 118, and the secondimaging device 126 determine which command signals are intended for thelight emitter(s) 116-1 and second light emitter 124, and relay the sameto the respective devices. Alternatively stated, the command signalstransmitted by computing device 102 may be intended to be relayed, viathe imaging devices, to the light emitters.

In non-limiting example embodiments, the computing device 102 isconfigured to provide command signals to the vehicle detector(s) 114,the imaging device(s) 118, and the second imaging device 126, which inturn determine or generate command signals for the respective lightemitter(s) 116-1 and second light emitter 124.

The vehicle detector(s) 114, the imaging device(s) 118, and the secondimaging device 126 may include onboard computing devices, whichimplement the functions of the vehicle detector controller 104, and thelight emitter controller 106, and receive command signals from theoccupant detector 110.

According to example embodiments, the below table shows an exampleconfiguration of the system 100:

Attribute Description Dimensions: Length ≈ 1.20 m Width ≈ 2 m Height ≈ 2m The sensors and light emitters on the system 100 can be fullyadjustable for height, and relative position and angle. The finalgeometry can be standardized. Weight: 60-70 kg Ground attachment: Thesystem 100 may use lockable casters. System 100 may have the ability tobe (semi-)permanently secured to a concrete pad. Physical security:System 100 can include weatherproofed latched panels, or waterproof keyaccess to internal components, or security screws for mountingcomponents such as the light emitter(s) 116, or various combinationsthereof. Power: 80-264 VAC 50-60 Hz single phase 1,000 W nominal LEDoutput −10% @ <90 VAC; −20% @ <85 VAC Power Conditioning: Optionalinternal surge protection and UPS depending upon the quality andreliability of the power supply. UPS intended to power the CPU andconnectivity, not the LED arrays. Safety Approvals: IEC60950-1 CB reportCSA 22.2No. 60950-1 UL60950-1 TUV, EN60950-1 SEMI F47 OperatingTemperature: −20° C. to +70° C. ambient. Units exposed to brightsunlight might exceed this range internally, and may be monitored.Additional fans or external coasting or other steps may be taken tosupport higher or lower temperatures. Weatherproofing: Light emitter(s)116 and vehicle detector(s) 114 are weatherproofed from the factory,imaging device(s) 118 are housed in weatherproof cases. For productionall cables will be weatherproofed, and vents/fans will be designed tosupport weatherproofed airflow and prevent ingress by dust, insects,etc. Network: 1 Gb Ethernet for remote and local access (formaintenance, etc.). 4 G LTE radio built in for use when Ethernet is notavailable. Network access control: Local access SSH via Ethernet, withsecurity controls implemented for both local and remote access. Range:System 100 may be configured for up to 9 m range. This range can beextended with additional IR illumination panel light emitter(s) 116.Image specification: Images are captured with a high-speed globalshutter industrial camera imaging device(s) 118 optimized for nearinfrared. 1920 × 1080 FHD uncompressed 120 Hz capture rate, supported bycustom IR LED arrays and controllers. Processor: Ruggedized passivelycooled Linux PC rated for outdoor temperatures, and placed inweatherproofed box. On board analysis: The Al models runs in real-timeon a local Linux server, analysing multiple images per vehicle (to moreaccurately detect occupancy when passengers are occluded by A, B or Cpillars. Local storage: The system 100 may include two 4 Tb SATA drivesconfigured in RAID 1 for robust local storage. Images for violatingvehicles (or other vehicles of interest) can be stored locally ifbroadband network connectivity is not available (of for training/testingdata capture).

FIG. 4 shows an example of method 400 for configuring a vehicledetection system.

At step 402, a distance from the target mounting location to a locationof a target lane is determined.

The target mounting location may be determined in part by the roadinfrastructure available in a particular location. For example, thesystem for vehicle occupancy detection 100 may be installed on aroadside post or other roadside fixture. In example embodiments, thetarget mounting location is based on a desired location of a mobilegantry which the system for vehicle occupancy detection 100 is attached.In some embodiments, for example, the target mounting location is basedon the traffic observed on the target lane, or the nature of the targetlane (e.g., the target location is placed near a high occupancy vehicle(HOV) lane).

A target lane can include one or more lanes of road which are expectedto have vehicle traffic and where vehicle occupation is desired to bedetermined. For example, the target lane may be a lane of a highway. Thelocation of the target lane is a location where a vehicle occupancy isdesired to be determined. For example, the location of the target lanemay be a location where a high occupancy vehicle (HOV) designated lanebegins.

At step 404, a width of the target lane is determined. The width of thetarget lane may be determined by manually measuring the lanes width. Inexample embodiments, the width of target lane is determined by taking animage of the target lane and processing the image to determine a width.

At step 406, a height of the ground at the target mounting location anda height of the location of the target lane are determined. For example,where the target lane slopes, and the location of the target lane isuphill of the target mounting location, the difference between therelative heights at the target mounting location and the location of thetarget lane is determined.

At step 408, a preferred mounting geometry is determined. In exampleembodiments, the preferred mounting geometry is determined in referenceto a base mounting geometry.

Referring now to FIG. 5A, a diagram of an example configuration 500 ofthe system 100, mounted according to a preferred mounting geometry, isshown. In example embodiments, the configuration 500 is a preferredmounting geometry, or the configuration 500 may be a base mountinggeometry.

Configuration 500 includes the computing device 102, the LiDAR vehicledetector(s) 114, the passenger light emitter(s) 116-1, the passengercamera imaging device(s) 118, the plate camera second light emitter 124,and the plate camera second imaging device 126 of system 100 connectedto a mounting device 506. The mounting device 506 may be a variety ofgeometries, and made of a variety of materials which allow for mountingof devices of system 100. For example, the shown mounting device 506 isshaped at least in part as having a support member 506-1, a firstattachment member 506-2, a second attachment member 506-3, and a thirdattachment member 506-4. Some or all parts of the mounting device 506may be a mobile gantry, or a roadside fixture. For example, the mountingdevice 506 may include as support member 506-1 the roadside fixture, andthe first attachment member 506-2 and the second attachment member 506-3may be metal supports passing through the roadside fixture. In anothernon-limiting embodiment, the system 100 may also be deployed in a securemobile trailer (not shown) that may be quickly moved from one locationto the next for rapid adhoc applications. In example embodiments, themounting device 506 includes lockable casters for attaching to thevarious constituent elements or the ground 504.

Network and power interface 502 carries out the functions of the networkswitch 230 and the power supply 232 of FIG. 2 . Network and powerinterface 502 is similarly connected to mounting device 506.

In the shown embodiment, the LiDAR vehicle detector(s) 114 is shown asbeing connected to third attachment member 506-4 0.55 [m] above theground 504.

The passenger light emitter(s) 116-1, and the passenger camera imagingdevice(s) 118 are shown as being connected to the first attachmentmember 506-2 1.45 [m] above ground 504. The passenger light emitter(s)116-1 are separated by a horizontal distance of 0.5 [m] from center tocenter along the first attachment member 506-2. The passenger lightemitter(s) 116-1 is also shown as 0.4 [m] horizontally distant from afirst end of the first attachment member 506-2, while the passengerlight emitter(s) 116-1 is shown as being 0.9 [m] horizontally distantfrom the first end of the first attachment member 506-2.

Plate camera second light emitter 124, and plate camera second imagingdevice 126 are connected to the second attachment member 506-3 at adistance of 2 [m] above the ground.

In example embodiments, installing the system 100 may be rapid as aresult of the modularity and fewer amount of parts of system 100, ascompared to multi-imaging device systems. For example, in someembodiments the system 100 may be deployed in less than one hour.

FIG. 5B is a perspective view of an example system for vehicle occupancydetection, in accordance with example embodiments;

The example system 514 shown in FIG. 5B has a configuration of the LiDARvehicle detector(s) 114, the passenger light emitter(s) 116-1, thepassenger camera imaging device(s) 118, similar to configuration 500, inthat the light emitter(s) 116-1 are upstream of the passenger cameraimaging device(s) 118, and the LiDAR vehicle detector(s) 114 is belowsaid components.

The vehicle detector may be upstream of the imaging device if thevehicle detector produces only a one-dimensional lateral measurement ofthe distance between the station and the vehicle. Examples of suchsensors are 1D laser range finders and under-road-pavement sensors(fiber optics, etc.) that record passage of the vehicle at a certainlateral distance from the station, without giving information about thelongitudinal position of the vehicle. Other vehicle detectors such as 3DLiDARs do not need to be necessarily placed upstream of the imagingdevice relative to the traffic direction. Such sensors can be placed indifferent locations as the physical and geometrical placement plays aless important role, relative to the role of the 3D perception processthat detects and tracks the vehicle, and triggers images by the imagingdevice.

In addition, system 514 includes a concreate pad platform 512, to whichmounting device 506 is attached, securing the system in a particularlocation. The platform 512 is shown to be about 2 meters long and 1meter wide. The height of the platform 512 can be anywhere between 0 to60 centimeters from the road surface (compared to the lane ofinterest).The platform 512 is preferably parallel to the lane ofinterest and has a flat and level surface 516.

In example embodiments, the platform 512 is a heavy steel structure, orother structure capable of fixing the system 514 to a particularroadside location (i.e., a ground surface adjacent to lanes of a road,which can include a shoulder, a service lane, median strip, orotherwise). A roadside location can also include road portions notdirectly adjacent to the road segments used for vehicle travel. Forexample, a roadside location includes, in example embodiments, a roadverge next to a road shoulder.

A small electrical cabinet (not shown in the picture) can be installedon surface 516 between the two posts of the structure 506 or elsewhere.According to some embodiments, the cabinet receives power from roadinfrastructure.

FIG. 5C is a perspective view of the system 514 of FIG. 5B includinganother imaging device, in accordance with example embodiments.

In the shown embodiment, the vehicle occupancy detection system 514 isconnected with, or controls, an imaging assembly 516 including animaging device and LI DAR unit mounted above the road. The imagingdevice and the LIDAR are pointed towards the road so that the imagingdevice captures images of the license plate of a vehicle as it drives inthe direction of traffic. The imaging assembly 516 may, for example, belocated 10 to 14 meters upstream of the vehicle occupancy detectionsystem 514, and can be installed on light poles or other roadsidefixtures which overhand traffic.

FIG. 5D shows a photograph of the system 514 of FIG. 5B, in accordancewith example embodiments.

FIG. 6A is a top view of an example system 602 for vehicle occupancydetection, in accordance with example embodiments.

Vehicle 604 is shown travelling in an expected direction of vehicletravel 606 (hereinafter referred to as direction 606), upstream of thevehicle detector(s) 114 and the imaging device(s) 118 in this exampleembodiment. Direction 606 is shown as being parallel to the lane marker620, in accordance with a typical vehicle direction travelling along alane.

Imaging device(s) 118 is shown having a horizontal field of view (e.g.,defined by edges 608A and 608B), and is pointed in a direction CAdefined by a yaw angle (ψ) of 15 degrees from an axis 610 perpendicularto lane marker 620. The yaw angle may be configured depending on theexpected speed of the vehicle travelling along the road. For example,the yaw angle may be decreased where traffic is expected to be slower,ensuring consistency between installations which have high traffic andlow traffic. In example embodiments, the yaw angle of the imagingdevice(s) 118 is fixed, and the patterns during which the light emitter(not shown in FIG. 6A) and imaging device(s) 118 emit light, and capturelight, respectively, is varied. The field of view of imaging device(s)118 may depend on the installation environment or intended use, and inexample embodiments, the field of view is 30 degrees. The yaw angle mayincrease the accuracy of the vehicle detection system by forcing imagesto contain certain perspectives, which, when multiple consecutive imagesare captured of vehicle 604 traveling at different speeds are capturedat said perspectives, contributes to all front and rear occupants of thevehicle 604 being visible in the captured images.

In some embodiments, the processor can compute the yaw angle. In someembodiments, there are multiple images from the vehicle so thatoccupants are seen from different perspectives as the vehicle travelshorizontally across the field of view, which can be reflected inextracted data from the images. A camera may have a large horizontalfield of view and the system may be able to achieve a good amount ofchange of perspective by taking multiple successive images as thevehicle is traveling from one end of the horizontal field of view to theother, even with a zero yaw angle. However, having a large of a field ofview may not always possible or favorable for some other reasons. Usinga nonzero yaw angle may accentuate the change of perspective within alimited horizontal motion of the vehicle in the field of view.Accordingly, the system computes data corresponding to change ofperspective, and, in some embodiments uses the yaw angle” as an examplemetric.

The vehicle detector(s) 114 is pointed in the direction of axis LA,which is perpendicular to direction 606. The vehicle detector(s) 114may, similar to imaging device(s) 118, be positioned with a yaw angletowards incoming traffic. Various positions of the vehicle detector(s)114, relative to imaging device(s) 118 are contemplated. The vehicledetector(s) 114 may be relatively close to the imaging device 114, or inexample embodiments, the vehicle detector(s) 114 may be much furtherupstream of the imaging device(s) 118 to account for the expected speedof the incoming traffic. In some embodiments, the vehicle detector(s)114 are not upstream relative to the traffic, and are in otherlocations.

Both the imaging device(s) 118 and the vehicle detector(s) 114 arelocated a distance 618 from an expected position of the vehicle. Thedistance 618 may be determined based on the geometry of the lane whichis being observed. For example, lanes vary in width, the location of thesystem 602 may be located further than the road in certaininstallations, and so forth. In example embodiments, the distance isdetermined by: L=ρ cos α, where L is the distance 618 in a directiondefined by axis 610, where ρ and α are shown in FIG. 6D. In exampleembodiments, the distance 618 is 5 meters. In example embodiments, thedistance from the imaging device(s) 118 to the expected position ofregion of interest 614 (e.g., the distance from the imaging device(s)118 to the middle of the lane) may be different from the distance fromthe vehicle detector(s) 114 to the expected position of region ofinterest 614 (e.g., the distance from the vehicle detector(s) 114 to themiddle of the lane).

Once the desired distance 618 is determined, the system 602 may be fixedwith this distance (e.g., secured to a concrete pad). Similarly, thedistance 616, in direction 606, between the imaging device(s) 118 andthe vehicle detector(s) 114 may be fixed after installation. In exampleembodiments, the distance 616 is 30 centimeters.

In FIG. 6A, the region of interest 614 is shown in part in the field ofview of imaging device(s) 118, and the region of interest 612 is not.Imaging device(s) 118 does not capture an image of vehicle 604 in FIG.6A as the vehicle 604 has not been detected by the vehicle detector(s)114.

In FIG. 6B, vehicle 604 has advanced in the direction 606 and the regionof interest 614 is shown directly in the line of sight of imaging deviceaxis CA. When the vehicle 604 is detected, the imaging device(s) 118 canbe activated or controlled (via hardware or software) to take one ormultiple images of the vehicle 604 in this instance.

In FIG. 6C, the region of interest 612 is prominently in the line ofsight of imaging device axis CA. Imaging device(s) 118 may be configuredto capture an image at this relevant distance instance. A region ofinterest is in a field of view when the feature reflects light towardsimaging device(s) 118 in a direction such that it is captured by theparticular configuration of imaging device(s) 118. For example, theremay be a region of interest that is not in the field of view of imagingdevice(s) 118 as light reflected from said feature cannot travel throughvehicle 604 and be captured by imaging device(s) 118.

FIG. 6D is a rear view of the example system 602 for vehicle occupancydetection of FIG. 6A. In contrast to imaging device(s) 118, the vehicledetector(s) 114 is shown as having a pitch angle α relative to axis 626a distance h_(L) to the ground 628. Vehicle detector(s) 114 is aimed indirection LA towards a point 624 horizontally further from the vehicledetector(s) 114 relative to an expected position of vehicle 604. Inexample embodiments, where ground 628 is flat, the pitch angle α is 5.25degrees and the point 624 may be approximately 10 meters away fromvehicle detector(s) 114. In this way, vehicles travelling closer to thevehicle detector(s) 114, for example at a distance p along axis LA, willinterfere with light travelling along axis LA and reflect light tovehicle detector(s) 114. In example embodiments, the vehicle detector(s)114 is positioned such that distance h_(L) is larger than some vehicles'wheel wells, providing a more accurate reading of whether a vehicle ispassing by. The distance h_(L) is approximately 90 centimetres accordingto some embodiments.

Imaging device(s) 118 is positioned a distance h_(C) above the ground628. In example embodiments, the distance h_(C) is 145 centimeters,which may be a distance of an expected height of an average car to theground 628. Imaging device(s) 118 has a line of sight CA which isparallel to the ground 628 in this example embodiment, however imagingdevice(s) 118 may have various pitch positions.

Referring again to step 408 in FIG. 4 , the preferred mounting geometrymay be determined in reference to the base geometry 500. For example,the preferred mounting geometry may include maximal variation for eachof the constituent elements of a base geometry. Continuing the example,the preferred mounting geometry may be constrained to have constituentelements placed within 20 cm of the configuration 500, and haveorientations (e.g., pitch, yaw, and roll) within 10° of theconfiguration 500. Advantageously, determining a preferred mountinggeometry based on a base mounting geometry may allow for a largervariation of configurations which provide accurate results, reducing theneed for meticulous calibration of the constituent elements of system100.

At step 410, the preferred geometry is output. The output preferredgeometry may be displayed on a display, allowing for a visual referencefor a technician to mount the vehicle occupancy detection system. Theoutput preferred geometry may be a geometry which enables the imagingdevice(s) 118 to capture more than one image of the detected vehicle.The system can store the output preferred geometry in memory, ortransmit the output preferred geometry to another system or component.

At step 412, the zoom of the imaging device(s) 118 may be adjusted inaccordance with the output preferred geometry. For example, thecomputing device 102 may be engaged to monitor whether the installationsatisfies the output preferred geometry. Continuing the example, wherethe computing device 102 determines that an imaging device(s) 118's zoomis not satisfactory, the display may display a notification includinginstructions required to adjust the imaging device(s) 118's zoom.

According to some example embodiments, step 412 includes the system 100operating in a diagnostic mode for a period of time until the systemdetermines that the installation satisfies the output preferredgeometry. For example, the output preferred geometry may be provided tothe occupant detector 110, which determines whether the preferredgeometry has been complied with after installation. In exampleembodiments, the output preferred geometry includes an indicator of theimaging device(s) 118's zoom, which may be continually monitored.

The system 100 may be modular, and the constituent elements may beattached to a mounting device (e.g., mounting device 506) separately,allowing for rapid deployment and set up.

The system 100, once mounted, may not require further training of theoccupant detector 110 in order to detect occupants. Alternativelystated, the occupant detector 110 may be pre-trained to work with thepreferred mounting geometry without additional training or adjustmentsto the machine learning model stored thereon.

FIG. 7 is a perspective view of a further example system 702 for vehicleoccupancy detection, in accordance with example embodiments. System 702,similar to system 514, includes light emitter(s) 116-1, and imagingdevice(s) 118-1 mounted on top of a gantry connected to a concrete pad512. The light emitter(s) 116-1, and imaging device(s) 118-1 arepositioned approximately 1.5 meters above ground to see overtop of theconcrete barrier 710. In example embodiments, the system 702 ispreferably positioned between approximately 4 to 8 meters from lane 620measured perpendicular relative to the direction of travel, on a roadportion adjacent to the nearest lane 620 and alternatively referred toas a roadside.

System 702 further includes light emitter(s) 116-3 and 116-4, andimaging device(s) 118-2 for vehicle occupancy detection positioned afurther distance above the ground 714, relative to light emitter(s)116-1, and imaging device(s) 118-1. Whereas light emitter(s) 116-1, andimaging device(s) 118-1 are positioned to capture images of vehiclestravelling in the first lane FL of traffic (e.g., based on their heightabove the ground and their pitch), the light emitter(s) 116-3 and 116-4,and imaging device(s) 118-2 are positioned (e.g., based on their heightabove the ground 714, and their pitch) to capture images of vehiclestravelling in the second lane SL of traffic. For example, in the shownembodiment, the light emitter(s) 116-3 and 116-4, and imaging device(s)118-2 are positioned approximately 2 meters above the ground 714, andimaging device(s) 118-2 is pitched downward approximately 10 degrees,with a yaw angle of 15 degrees.

Vehicle detector(s) 114 is positioned above both light emitter(s) 116-1,and imaging device(s) 118-1 and light emitter(s) 116-3 and 116-4, andimaging device(s) 118-2. In example embodiments, a plurality of vehicledetectors 114 are used to determine, respectively, whether a vehicle ispassing in each respective lane. In example embodiments, plurality ofvehicle detectors 114 are used to detect passing vehicles in eitherlane. In example embodiments, the vehicle detectors 114 are capable ofscanning a wide horizontal field of view (e.g., at least 120 degrees)and a reasonable vertical field of view (e.g., at least 30 degrees).

In the shown embodiment, the vehicle detector(s) 114 is positioned witha yaw angle such that it is able to detect vehicles relative to thesystem 702 (e.g., vehicles 604 and 708). For example, the vehicledetector(s) 114 may detect vehicles approximately 15 to 20 meters beforethey are in the field of view of the respective imaging devices. Inexample embodiments including a plurality of vehicle detectors 114, eachvehicle detector may be respectively positioned to detect vehicles atdifferent distances.

The vehicle detectors may be 2D or 3D LIDAR units capable of capturingmultiple readings of distances in multiple directions. For example,vehicle detector(s) 114 in the shown embodiment emits a plurality oflight 704 (e.g., infrared light) towards both the first lane FL and thesecond lane SL. A potential advantage of using a 2D or 3D LIDAR vehicledetector(s) 114 that is capable of capturing a point cloud from movingvehicles compared to the single laser beam range measurement isincreased robustness to dust particles and precipitation. Whilemeasurements from a single laser beam can easily get contaminated bynoise, an entire point cloud of measurements from a vehicle isstatistically more robust to noise. Also since the vehicles (e.g.,vehicles 604 and 708) can be detected before they are within the fieldof view of an imaging device, the more robust detection of passingvehicles may provide for more precise adjustments of the pattern betweenthe light emission, image capture and vehicle detection. In exampleembodiments, the more precise estimation allows for detecting thevehicles a greater distance from the system 702, and allows greaterfiltering windows (i.e., the use of larger windows of time betweendetection of the vehicle and capturing an image of the vehicle) withoutrisking detecting the car too late.

Imaging devices 118-3 and 118-4 may be used to capture images of thefront and rear license plates. For example, in the shown embodiment,imaging device(s) 118-3 is at a yaw angle which points in the direction606 to capture rear license plates.

FIG. 8 is a flowchart of an example of method 800 for vehicle occupancydetection, in accordance with example embodiments.

Method 800 may be implemented by the occupant detector 110, for example,or by a remote computing device.

At step 802, a detection signal is received from the vehicle detector(s)114. In example embodiments, the detection signal includes a detectedspeed of the detected vehicle.

In example embodiments, as a result of the geometry of the installationof system 514 (e.g., the yaw angle ψ of imaging device(s) 118, thehorizontal field of view of the camera (e.g., defined by the edges 608Aand 608B), distance between the imaging device(s) 118 and the vehicledetector(s) 114, etc. shown in FIG. 6A), the system ensures that whenthe vehicle 604 is detected, the imaging device(s) 118 is triggeredinstantly, and the entire vehicle 604 is within the camera's horizontalfield of view (e.g., FIG. 6B).

At step 804, a command signal is transmitted to the light emitter(s) 116to emit light according to a first pattern for a first time window. Inexample embodiments, the first pattern is determined by the speed of thevehicle as detected by the vehicle detector 114. According to someembodiments, for example, the first pattern is a preconfigured frequencybased on the configuration of the system 100. Continuing the example,the preconfigured frequency may be based on the detection distance, thelatency associated with vehicle detection, and the operating frequencyof the imaging device(s) 118.

In an illustrative embodiment, for traffic where vehicles are expectedto be travelling with speeds around 80-140 km/h, the frequency can be 90pulses per second. This frequency can provide for 5 sufficient qualityimages of passing vehicles.

In example embodiments, once the vehicle 604 is detected, the vehicleposition, direction of travel and speed is tracked using a trackingapproach, such as a Kalman filter. The estimation of position and speedof the vehicle 604 can then be used to trigger, for example, the licenseimaging device(s) 118-3 of FIG. 7 (e.g., when the vehicle 604 is about10-14 meters upstream of system 702), and then trigger the imagingdevices 118-1 and 118-2 multiple times at optimal places to takemultiple shots for occupancy counting. The tracking approach keepstracking the vehicle until it passes the system, and when it is 10-14meters away it triggers the imaging device(s) 118-4 to capture images ofthe rear license plate if necessary. The detection and tracking of carsin multiple lanes (at least lane 1 and 2, and possibly the shoulder laneor lane 3) can happen simultaneously in a perception software systemwhich can be implemented by computer 102.

In example embodiments, the occupant detector 110 may receive ambientcondition information from the ambient condition sensor 130, anddetermine an optimal configuration for the light emitter(s) 116 based onthe received ambient condition. The optimal configuration is thentransmitted along with the control signals. For example, based onreceived ambient conditions, the occupant detector 110 may determinethat the light emitter(s) 116 intensity should be increased, andtransmit control signals reflecting same.

The first time window may be, similar to the first pattern, dynamic orpreconfigured.

At step 806, the command signal is transmitted to the imaging device(s)118 to capture images according to a second pattern associated with thefirst pattern, for a second time window associated with the first timewindow. The second pattern is associated with the first pattern of thelight emitter(s) 116 so that the imaging device(s) 118 captures thelight emitted by the light emitter(s) 116 (e.g., the imaging device(s)118 captures images after light has been emitted). In exampleembodiments, the second pattern may be based on the latency associatedwith the light emitter(s) 116 emitting light, and the latency associatedwith the command signal reaching the imaging device(s) 118. In anon-limiting example embodiment, the imaging device(s) 118 may beconfigured to capture successive high-speed snapshots (e.g., 5 images)of the detected vehicle as it passes the system 100.

In example embodiments, the transmitted command signal includesconfiguration signals for adjusting the imaging device(s) 118'sacquisition parameters (e.g., the number of pictures taken for eachvehicle, imaging device exposure time, imaging device frame rate, gainsettings, focal length, etc.) based on the ambient conditions in orderto maximize the quality of the image acquisition, which may in turn leadto higher overall accuracy. The ambient conditions may be received bythe computing device 102 from the ambient condition sensor 130, from anexternal ambient condition data service provider, or otherwise. Forexample, at higher detected vehicle speeds, the imaging device(s) 118may capture images at a greater frequency (i.e., a higher FPS) orimaging device(s) 118 may begin capturing images more rapidly inresponse to a vehicle being detected (e.g., using a shorter filteringalgorithm window). Conversely, at lower detected vehicle speeds, imagingdevice(s) 118 may reduce the frequency of image capture (i.e., the FPS)or increase the algorithm window size.

In example embodiments, the command signal transmitted to the imagingdevice(s) 118 is configured to avoid lens distortion effects associatedwith the vehicle 604 being too close to the margins of the capturedimages (e.g., too close to edges 608A and 608B). The system 100 maycompute a speed of travel, and align the command signal to captureimages of the vehicle without the region of interest being within athreshold of the edges 608A and 608B. For example, based on an averagevehicle length of 4.5 to 5 meters, the system 100 can estimate the speedof traffic as multiple cars pass by the vehicle detector(s) 114 (e.g.,each car passage will register similar to an inverted square pulse, andthe time-length of the pulse assuming a nominal length of vehicles, canbe used to estimate speed of traffic). Alternatively, the system 100 canbe integrated with other systems such as toll bridges that monitortraffic and estimate vehicle speeds, and system 100 may estimate alikely position of the vehicle based on said data to adjust the durationand frequency of operation of the imaging device(s) 118 (e.g., thefiltering window length and image acquisition speed (FPS)).

The second time window may be, similar to the first time window, dynamicor preconfigured.

Optionally, at steps 808 and step 810, the second light emitter 124 andthe second imaging device 126 may be configured to, respectively, emitlight and capture images associated with the rear of the detectedvehicle. The images associated with the rear of the detected vehicle maybe analyzed to determine a license plate number of the detected vehicle.

At step 812, the occupant detector 110 receives the captured images fromthe imaging devices, and determines a vehicle occupancy.

In some embodiments, the occupant detector 110 determines a first regionof interest of the vehicle in each of the plurality of captured images,a second region of interest of the vehicle, and determines the number ofvisible occupants in the each region of interest image as the vehicleoccupancy. For example, the occupant detector 110 may be trained todetect occupants based on expected positions within the detected vehicle(e.g., it is more interested in the location of a vehicle above or nearto a seat, as opposed to spaces between seats). Continuing the example,the occupant detector 110 may then use as a vehicle occupancy themaximum number of determined occupants for each window (e.g., whether 1,2 or 3 occupants are visible in the window), combining the results frommultiple images into a single result (e.g., a simplemax_per_window_acrossallimages approach).

The occupant detector 110 may determine the vehicle occupancy in partbased on determining a rear and a front occupancy. For example, theoccupant detector 110 may separately determine the amount of occupantsvisible in each seating row of a detected vehicle. This approach mayhave the advantage of simplifying the system 100, introducingredundancy, and in turn improving accuracy and reducing overall cost ofthe system.

According to example embodiments, the vehicle occupancy may bedetermined as the most likely number of occupants based on each of therespective number of visible occupants in each image processed. Forexample, where the occupant detector 110 determines differing amounts ofoccupants in each of the images of the detected vehicle, it may beconfigured to determine the vehicle occupancy as the most commonlyoccurring number of occupants across images, or the number of occupantsdetected by the images which are side views of the vehicles, and soforth.

In further example embodiments, the occupant detector 110 uses anoccupant model to determine the number of vehicle occupants. Forexample, the occupant detector 110 may be trained to fit an occupantmodel to determine a most likely model which first all images associatedwith the detected vehicle.

The occupant detector 110 may normalize the images prior to processingsame. For example, the occupant detector 110 may normalize the images sothat the vehicle is the same size in each image, or normalize the imagesso that the effect of ambient conditions is consistent across images(e.g., images with strong glare may be filtered to reduce glare). Theimages which have normalized vehicles may be normalized based onoccupant detector 110 determining respective normalization parametersbased on the vehicle speed and the first pattern and the first timewindow.

In some embodiments, for example, the occupant detector 110 may generateand use a normalized vehicle model populated with each of the pluralityof images processed with the respective normalization parameters tonormalize the vehicle across the captured plurality to images.

The occupant detector 110 may be configured to discard images which donot include the detected vehicle (e.g., false positive triggers). Forexample, where the vehicle detector(s) 114 detects a vehicle, a separatevehicle detector (not shown), such as a machine learning modelconfigured to detect vehicles in images, as opposed to using LiDAR, maydetermine whether the captured images include a car (as opposed to ananimal, etc.).

The occupant detector 110 may detect a vehicle occupancy of a detectedvehicle by normalizing detected two or more vehicles in the plurality ofimages relative to one another. For example, where there are fivesuccessive images include multiple detected vehicles, the occupantdetector 110 may be configured to enlarge the portions of the imageswith the respective detected vehicles so that they are the same size.

According to some embodiments, for example, the occupant detector 110may further normalize images with respect to the ambient conditions. Forexample, where the direction of the sun is detected (e.g., via thedirection of sunlight intensity), the images wherein the vehicle isincident with more powerful sunlight may be filtered to mimic conditionsin other images where the sunlight is weaker.

Optionally, at steps 814 and 816, the occupant detector 110 may beconfigured to respectively generate and transmit a report. In exampleembodiments, the report is generated and transmitted in response todetermining a vehicle occupancy outside of a threshold. The report mayfor example be generated for and transmitted to a tolling agency (notshown), which tolls vehicles in response to the occupancy detectionoutside of the threshold.

In example embodiments, the threshold is based on a determination ofwhether a vehicle is in violation of existing vehicle occupation lawover a confidence interval. For example, where the detected vehicle isin a high occupancy vehicle (HOV) lane which requires more than 3occupants, the threshold may be whether the system 100 is more than 90%confident that there are more than three occupants. Further discussionof the confidence interval is discussed below.

The report may include a date, time of day, estimated speed, vehicletype, lateral position of vehicle in lane, front occupancy, rearoccupancy, overall occupancy, front occupancy confidence, rear occupancyconfidence, overall occupancy confidence, and a license plate of thevehicle (detected from the images captured by the second imaging device126), all of which may be extracted from the images received by theoccupant detector 110. The report may include health monitoringinformation on all of the sensors and hardware components of system 100.

The report may be stored in a standard SQL database allowing forinterfacing with well-known application program interfaces (APIs) toquery the data and generate any desired report via SQL queries.

The report may include the one or more captured images and metadataassociated with the one or more captured images of the detected vehicle,and bounding boxes representing the detected occupants.

In example embodiments, the report is a report which describes, forexample, detected vehicles weaving between lanes (e.g., where acalculated speed and position of the vehicle is outside of the expectedlane markers) and/or stunt driving (e.g., erratic behaviour of theregion of interest—such as high speeds, dangerous proximity to othervehicles, etc.) and documenting such unlawful driving behavior forpurpose of law enforcement.

According to some embodiments, the report may include informationgleaned from monitoring traffic over multiple lanes over different hoursof the day, different days of the week and different seasons, andextracting useful information and statistics about road usage. Forexample, the report may provide comprehensive statistics about whichlanes are most dangerous, which lanes appear to have potholes (e.g.,consistent weaving of lanes in a particular location), drivingcharacteristics and how they change in response to the environment(e.g., tracking the performance of a snow removal contractor over time)and so forth.

FIGS. 15A and 15B show an example of vehicle weaving and an anti-weavingfeature of the system 100.

As shown in FIG. 15A, the system 100 can be made to detect if a vehiclepassing in front of it is trying to weave away into farther lanes in anattempt to avoid having its occupancy counted. The anti-weaving featureof the system 100 is useful to ensure usage of the entire system for HOV(high occupancy vehicle) lane and HOT (high occupancy toll) lane usecases.

As shown in FIG. 15B, an example implementation can use the same vehicledetector sensor of the system 100, such as for example a sensor that is3D LiDAR. The example in FIG. 15B shows 3D visualizations observed bythe vehicle detector sensor that can be utilized to observe thetrajectory of the vehicle motion on the road and detect if a lane changeis occurring in the lateral direction before and after the stationlongitudinally. As an alternative to 3D LiDAR in the event a station isequipped with only a 1D laser range finder or any other vehicle detectorwith limited capabilities, for example, the system 100 can have anadditional camera installed higher up and a large field of view toobserve this vehicle motion.

The system can compute a trajectory of the vehicle motion on the roadand detect if a lane change is occurring in the lateral direction beforeand after the station longitudinally. The system can use 3D LiDAR or addan additional camera installed higher up and a large field of view toobserve this vehicle motion.

In some embodiments, there is provided a system for detecting occupancyof a vehicle travelling in an expected direction of travel along a road.The system has a first roadside imaging device positioned on a roadside,having a first field of view of the road, the first field of viewincident on a side of the vehicle when the vehicle is on the road withinthe first field of view. The system has a first roadside light emitteremitting light towards vehicles in the first field of view. The systemhas a roadside vehicle detector. The system has a processor, incommunication with a memory, configured to: receive a signal from theroadside vehicle detector indicating that the vehicle is within orproximate, relative to the expected direction of vehicle travel, to thefirst field of view; command the first roadside light emitter to emitlight according to a first pattern for a first duration; command thefirst roadside imaging device to capture images of the side of thevehicle according to a second pattern associated with the first pattern,during a second duration associated with the first duration; receive thecaptured images of the side of the vehicle from the first roadsideimaging device; compute a vehicle occupancy of the vehicle by, in eachof the captured images: determining one or more regions of interest ofthe vehicle in each of the captured images; determining the vehicleoccupancy as a number of visible occupants in the one or more regions ofinterest; and determining a most likely number of occupants based oneach determined vehicle occupancy. The system can transmit the vehicleoccupancy to a monitoring system.

FIG. 9 is a flowchart of an example method to complete step 812 of FIG.8 for detecting occupants in images, in accordance with exampleembodiments.

At block 902, images are received, for example by the occupant detector110. In some embodiments, an image may be received, or in someembodiments two or more images may be received.

Capturing more than one image can enable the system to extract moreinformation from multiple images and can help avoid obstruction ofoccupants by the vehicle window frames or obstruction of farther sittingoccupants by closer sitting occupants. However, in some cases even oneimage can be sufficient. Multiple images may achieve higher performanceand robustness, but capturing and processing an image may also providesufficient data in some embodiments.

At block 904, each image is processed to determine the pixels associatedwith a window of a vehicle. For example, the occupant detector 110 mayimplement an SST detector, trained to identify a vehicle in an image.Where no vehicle is detected, the occupant detector 110 may record thisas an instance of no occupants.

At block 906, a region of interest is determined for each image. Inexample embodiments, this block is performed simultaneously with block904. In example embodiments, one or more regions of interest areidentified, such as a front and rear side window (e.g., region ofinterests 612 and 614). Where a region of interest is not detected, theimage is discarded, or the occupant detector 110 may record this as aninstance of no occupants.

The license plate recognition can be done on the front side or the rearside, or both sides. Lighting conditions and country/province ofoperation (e.g., depending on requirement a front plate on vehicles) arefactors to consider.

Optionally, at block 908, the images are cropped so that only pixels inthe region of interest are referred to for occupant detection. Inexample embodiments, this may allow for a more efficient occupantdetector capable of running on legacy systems with limited computingresources.

At block 910, the cropped image(s) are processed with the occupantdetector 110 using a classifier to identify a number of occupants withinthe region of interest. For example, the classifier may be a single shotclassifier SST trained to identify individuals in pixels.

At block 912, the vehicle occupancy is determined based on theclassified number of individuals identified in block 910. For example,the occupant detector 110 may average the number of occupantsidentified.

In example embodiments, where the region of interest includes a frontand a rear side window, the occupant detector 110 is configured to, (1)determine the amount of individuals present in the rear and front sidewindows, and (2) average, over the plurality of images, the number ofdetected occupants in each of the rear and the front side windows.Continuing the example, if there are five images, and the followingnumber of occupants are detected in the rear side window in successiveimages: 2, 3, 2, 1, 2, the occupant detector 110 may determine thatthere are 2 occupants in the rear of the vehicle. A similar process maybe carried out for the front side window. In example embodiments, wherethe region of interest includes a front and a rear side window, theoccupant detector 110 is configured to count the number of occupantsidentified in each image for each of the front window and the rear sidewindow, and determines the vehicle occupancy as the sum of (1) themaximum number of detected individuals in the front side window, and (2)maximum number of detected individuals in the rear side window.

Referring now to FIGS. 10A to 10G, which each show an image of a vehiclewith various regions of interest shown, in accordance with exampleembodiments.

FIG. 10A shows an example visual representation wherein bounding boxeshave been accurately associated with four occupants in a single vehicle.FIGS. 10B-10D show example visual representations of multipleindividuals being identified in multiple vehicles across multiple lanes.FIG. 10E includes bounding boxes identifying an occupant despite tintedwindows. FIG. 10F shows an example visual representation wherein fourindividuals have been accurately identified in the first detectedvehicle, including an individual whose head is turned away from theimaging device. FIG. 10G shows an example visual representation whereinbounding boxes have been accurately associated with an occupant, andhave correctly not identified an animal as an occupant.

In example embodiments, the report generated by the occupant detector110 may include historical information about vehicle occupancy asdetermined by the system 100. For example, in the shown visualrepresentation of FIG. 11 , the occupant detector 110 outputs a reportwhich includes an interactive chart representing the average totalnumber of occupants detected over a period of time. Advantageously, suchreports may be used to determine road capacity, road usage, and changingtraveller compositions over time. In example embodiments, the occupantdetector 110 outputs various report information into a visual interfacecapable of being interacted with. For example, the occupant detector 110may output detection rates for storage or transmission.

The system 100 may be capable of achieving accuracy of detecting vehicleoccupancy at significantly higher rates than can be achieved by humanobservation.

In some embodiments, for example, once the report is generated andtransmitted to the tolling authority, or other third party, the system100 deletes all local storage of the plurality of images associated withthe occupancy detection.

In example embodiments, the system 100 may include one or more privacyfeatures to prevent the imaging data from being inappropriatelyauthorized. The computing device 102 may be configured to process theimage with the occupant detector 110 locally to prevent loss ofsensitive image data. The system 100 may store data (e.g., on database112) on hard drives that are encrypted, in addition to encrypting everyfile (image or otherwise) associated with the operation of the system100. In some embodiments, the computing device 102 may be configured to,prior to saving any image, detect faces in the images and blur beyondrecognition any detected faces. Any transmission of data originatingfrom within system 100 (e.g., command signals, images, etc.) may beencrypted prior to transmission, and any stored data within the system100 may be configured to be deleted after a data retention deadlinepasses.

In example embodiments, system 100 may be configured to save andtransmit only the images associated with deemed violators of vehicleoccupation rules, thereby further minimizing the scale possiblebreaches.

FIG. 12 is an architecture diagram 1200 of the system 100, according toexample embodiments.

In FIG. 12 , at step 1202, the vehicle detector(s) 114 detects andtimestamps LIDAR data at high frequency.

At step 1204, the system 100 processes the received LIDAR data with asignal processing algorithm to detect a passing vehicle with low latencyin one or more Lanes of Interest (Lol). In example embodiments, asdescribed, the signal processing techniques determine whether thedetected range changes.

At step 1206, the imaging device(s) 118 are activated to capture imagesand light emitter(s) 116 are activated (shown camera trigger 1218 asflash trigger 1220). In example embodiments, imaging device(s) 118 andlight emitter(s) 116 are activated simultaneously.

At step 1208, the computing device 102 detects features of interest inthe captured images. For example, computing device 102 may performmethod 900.

At step 1210 the computing device 102 determines the number of occupantsin the regions of interest of step 1208.

At step 1212, optionally, the computing device 102 may store all or someof the received and processed data. For example, the computing device102 may store the received images into database 112, including atimestamp and metadata (number of people, debugging data, cameraparameters, LIDAR triggering information, etc.).

At step 1214, the computing device 102 may transmit the stored data to aweb server, such as the web server of a system operator.

The web server, which may be a separate computing device 102, remote tothe computing device located on the roadside unit, may run in parallelto the roadside system (e.g., system 514) to access the latestacquisitions and inspect results from the system. The web server alsocan be used to tune configuration parameters such as shutter time,camera gain and desired Lanes of Interest for the roadside system.

According to example embodiments, the computing device 102 may determineat step 1204, or at any point after the vehicle detector(s) 114 hastriggered, whether the vehicle detector(s) 114 was correct indetermining a vehicle detection, referred to as trigger accuracy.Trigger accuracy may be an important aspect that determines overallperformance of system 100.

Trigger accuracy may be represented by the trade-off between two errortypes: false triggers (triggering when no vehicle is there typicallybecause of rain, dust or snow) and missed triggers (not triggering whena vehicle is in fact there).

The computing device 102 can be configured to reject false triggers(e.g., if no vehicle is present in the set of acquired images, saidimages are simply discarded), as false triggers can cause prematureaging of the system.

In example embodiments, the system 100 is trained to reduce both falsetriggers and missed triggers to a minimum. Example field test resultsare shown below:

Condition True Trigger (Day & Night) (100%-missed) False Trigger Idealweather 99%  1% Light rain/fog/snow 97%  4% Heavy 84% 11% rain/fog/snow

In example embodiments, the computing device 102 may determine at step1204 whether the occupant detector 110 was correct in determining avehicle occupation, based on the system's 100 ability to overcome darkwindows. For example, the system 100 may use ultra-high power narrowbandnarrow-field Infrared (IR) light emitter(s) 116 with matched imagingdevice(s) 118 camera sensor and filters. The light emitter(s) 116 mayuse a light wavelength that simultaneously maximizes penetration ofwindow tint and minimizes interference from the sun, and in exampleembodiments, the system 100 can have two large LED panel lightemitter(s) 116 capable of penetrating window tint at a distance of up to9 meters. The occupant detector 110 may review images to determinewhether the images include vehicles where a detection signal isreceived, and determine whether a vehicle is present in the images.

In example embodiments, the computing device 102 may determine at step1204 whether the occupant detector 110 was correct in determining avehicle detection, based on the system's 100 ability to distinguishbetween humans and other objects. For example, the occupant detector 110may be a deep neural network, trained on training data consisting ofover 250,000 examples specific to the vehicle occupancy detection (VOD)case, as well as millions of training images outside of the VOD contextfor further robustness. As a result, the occupant detector 110 may beable to distinguish human beings in some of the most difficult posescompared to animals or other objects. According to example embodiments,the use of infrared imaging may be able to distinguish human beings fromdolls as doll skin material may react differently to the infraredillumination to a degree sufficiently different compared to human skin.

In example embodiments, the computing device 102 may determine at step1204, or at any point thereafter, whether the occupant detector 110 wascorrect in determining a vehicle detection, based on the system's 100ability to detect curtains and possible obstructions (e.g., curtains,pants hanging, etc.) for further review. For example, the computingdevice 102 may be trained to, instead of detecting no occupants wherecurtains are shown, flag images with detected curtains for furthervalidation.

At step 1216, the computing device 102 may upload the stored data with adata upload service.

The computing device 102 may prompt a user to validate the occupancydetection in response to an image being flagged, or in response to asuspected trigger accuracy error. Referring now to FIG. 13 , an exampleuser interface 1300 for validating use occupancy is shown.

In the shown embodiment, user interface 1300 includes an image displaypanel 1302, an image display slide 1304, and image enhancement panels1306-1 and 1306-2.

The image display slide 1304 may be used by the user to control theimage displayed in the image display panel 1302. In the shownembodiment, five images are associated with an occupancy detection, andthe slider allows for changing the image display panel 1302 to any ofthe five images.

Each of image enhancement panels 1306-1 and 1306-2 may show an enlargedview of a portion of the image shown in image display panel 1302 foreasier viewing. In some embodiments, the image enhancement panels 1306-1and 1306-2 show the previous and subsequent image associated with theparticular vehicle object detection.

Validation may consist of receiving user input associated with any oneof occupant validation input 1308-1, occupant validation input 1308-2,occupant validation input 1308-3, and occupant validation input 1308-4(hereinafter the occupant validation inputs). User selection of theoccupant validation inputs can indicate the correct number of occupantsin the images shown in image display panel 1302. For example, userselection of the occupant validation input 1308-1 and occupantvalidation input 1308-2 can be indicative of 1 or 2 occupants, or morethan 3 occupants. In example embodiments, various numbers of occupantvalidation inputs are contemplated.

User selection of occupant validation input 1308-3, which isrepresentative that the image cannot be validated, can trigger thegeneration and display of a drop down menu which includes selectableelements for indicting the reason the image cannot be validated. In someembodiments, the drop down menu includes the following reasons: theimage was too dark, too much glare, the image was obstructed, and thetint was not overcome.

Occupant validation input 1308-4 may be used to cycle between occupancydetection. Exit element 1310 can be used to stop validation of theselected image.

In example embodiments, a further imaging device (not shown) is usedwith the system 100, which will provide image data used to validate thedetected vehicle occupancy. For example, imaging device(s) 118-3 of FIG.7 may be used as this further imaging device. Images captured by theimaging device may be used for monitoring and tuning purposes, andaccessed through interface 1300.

FIG. 14 is a schematic diagram of computing device 102, in accordancewith an embodiment.

As depicted, computing device 102 includes at least one processor 1402,memory 1404, at least one I/O interface 1406, and at least one networkinterface 1408.

Each processor 1402 may be, for example, any type of microprocessor ormicrocontroller (e.g., a special-purpose microprocessor ormicrocontroller), a digital signal processing (DSP) processor, anintegrated circuit, a field programmable gate array (FPGA), areconfigurable processor, a programmable read-only memory (PROM), or anycombination thereof.

Memory 1404 may include a suitable combination of any type of computermemory that is located either internally or externally such as, forexample, random-access memory (RAM), read-only memory (ROM), compactdisc read-only memory (CDROM), electro-optical memory, magneto-opticalmemory, erasable programmable read-only memory (EPROM), andelectrically-erasable programmable read-only memory (EEPROM),Ferroelectric RAM (FRAM) or the like.

Each I/O interface 1406 enables computing device 102 to interconnectwith one or more input devices, such as a keyboard, mouse, camera, touchscreen and a microphone, or with one or more output devices such as adisplay screen and a speaker.

Each network interface 1408 enables computing device 102 to communicatewith other components, to exchange data with other components, to accessand connect to network resources, to serve applications, and performother computing applications by connecting to a network (or multiplenetworks) capable of carrying data including the Internet, Ethernet,plain old telephone service (POTS) line, public switch telephone network(PSTN), integrated services digital network (ISDN), digital subscriberline (DSL), coaxial cable, fiber optics, satellite, mobile, wireless(e.g., Wi-Fi, WiMAX), SS7 signaling network, fixed line, local areanetwork, wide area network, and others, including any combination ofthese.

For simplicity only, one computing device 102 is shown but computingdevice 102 may include multiple computing devices 102. The computingdevices 102 may be the same or different types of devices. The computingdevices 102 may be connected in various ways including directly coupled,indirectly coupled via a network, and distributed over a wide geographicarea and connected via a network (which may be referred to as “cloudcomputing”).

For example, and without limitation, a computing device 102 may be aserver, network appliance, set-top box, embedded device, computerexpansion module, personal computer, laptop, personal data assistant,cellular telephone, smartphone device, UMPC tablet, video displayterminal, gaming console, or any other computing device capable of beingconfigured to carry out the methods described herein.

In some embodiments, a computing device 102 may function as a clientdevice, or data source.

In some embodiments, each of the vehicle detector controller 104, thelight emitter controller 106, the imaging device controller 108, theoccupant detector 110, and the second imaging device controller 122 area separate computing device 102. In some embodiments, the vehicledetector controller 104, the light emitter controller 106, the imagingdevice controller 108, the occupant detector 110, and the second imagingdevice controller 122 are operated by a single computing device 102having a separate integrated circuit for each of the said components, ormay be implemented by separate computing devices 102. Variouscombinations of software and hardware implementations of the vehicledetector controller 104, the light emitter controller 106, the imagingdevice controller 108, the occupant detector 110, and the second imagingdevice controller 122 are contemplated. In some embodiments, all orparts of the vehicle detector controller 104, the light emittercontroller 106, the imaging device controller 108, the occupant detector110, and the second imaging device controller 122 may be implementedusing conventional programming languages such as Java, J#, C, C++, C#,Perl, Visual Basic, Ruby,

Scala, etc. In some embodiments, these components of system 100 may bein the form of one or more executable programs, scripts, routines,statically/dynamically linkable libraries, or the like.

For the confidence interval, training of the occupant detector 110 mayinclude distinguishing between false positives and false negatives.

The occupant detector's 110 accuracy (alternatively referred to asperformance) may be assessed (for example, during training) usingcalculations of False Positive (FP) and False Negatives (FN), which mayvary depending on the application.

Simple 2-Stage Model—High vs. Low Occupancy

The most common objective of the occupant detector 110 is to distinguishbetween a high-occupancy vehicle (for example, a vehicle with 2 or moreoccupants) and a low-occupancy vehicle (for example, a vehicle with onlya driver—single occupant). The system performance can be expressed as a“confusion matrix” of 4 numbers (N1, N2, N3, N4). The 4 numbers in theconfusion matrices should be independent nonnegative integer numbers.The numbers do not need to add up to 100 & row-wise or column-wise. Aconfusion matrix can include the following:

TABLE 1 Confusion Matrix Predicted Occupancy 1 occupant 2+ occupantsActual 1 occupant TP FN occupancy 2+ occupants FP 1TN

20

The confusion matrix example shows the rates at which actual “x occupantvehicles” are identified as “y occupant vehicles” for all possiblecombinations of “x” and “y”. The confusion matrix (and therefore systemperformance) is completely characterized by two types of errors, namely(1) False Negatives (Top-right corner, red, “FN”): A low-occupancyvehicle is incorrectly seen as high-occupancy, in a high occupancyvehicle (HOV) context, this means the percentage of violators that aregiven a “free pass”, and (2) False Positives (Bottom-right corner, red,“FP”): A high-occupancy vehicle is incorrectly seen as low-occupancy. Inan HOV context, FP represents the percentage of honest road users thatare wrongfully ticketed.

The cells in the confusion matrix that represent correct guesses arerelated to the error rates as shown in Table 1. These errors are not aspecific quality of the system 100 but are rather the nature of thevehicle occupancy detection (VOD).

The system 100 may have the capability to adjust a relative weight ofthe FP errors and the FN errors before or during roadside deployment.Alternatively stated, the system 100 may trade one type of error foranother depending on the configuration. The system 100 may be adjustedbased on a wide range of FN and FP variations. For example, according tosome embodiments, the system 100 can be configured such that both typesof errors (wrong tickets and free passes) are given equal importance. Inother example embodiments, the system 100 can be setup in a mode wherewrong tickets are given more importance and reduced at the expense ofincreased free passes. Multiple variations of relative weighing of theFN and FP errors are contemplated.

In example embodiments, the system 100, instead of determining themutually exclusive “low-occupancy” or “high-occupancy” may output (e.g.,in a report) a continuous probability/confidence that can be normalized.The closer the confidence/probability value can provide an indication ofhow confident the system 100 is about the detected vehicle having lowoccupancy. A road operator may select to flag or ticket or take otheractions with respect to all vehicles above the threshold confidence. Ifthe threshold is normalized to represent sensible/meaningful numbers,the threshold can be determined or configured by operators of the system100.

In example embodiments, the degree of confidence (or confidence value)can be discretized, such that various confidence values are associatedwith various pre-set use cases, or any number of operating modes may beconfigured by the operator. For example, a confidence threshold can beconfigured for a first mode of operation of system 100 in order toticket individuals. The greater the confidence threshold, the less riskthe tolling operator will have on creating a false positive. However,the tolling operator will operate with a higher chance of missingviolations with such a high confidence threshold.

Some example embodiments of system configuration for relative weight ofthe FP errors and the FN errors (alternatively referred to as modes) arefurther described below:

Example Mode A

In example mode A, the system 100 is deployed such that the occupantdetector 110 is trained that wrongfully identifying honest users has anequal importance to giving violators a free pass. For example, thisconfiguration may be used in an area where a fair amount of bothlow-occupancy and high occupancy vehicles are expected. In this mode,both FP and FN errors are treated as equally important.

TABLE 2 Mode A-Confusion Matrix for the system where both error typesare considered of equal importance Predicted Occupancy 1 occupant 2+occupants Actual 1 occupant 908 92 occupancy 2+ occupants 88 912

In an example configuration in accordance with mode A, the test resultsare shown in Table 2 above, and there is a computed chance that thesystem will make an error and give either a “free pass” or a wrongticket.

Example Mode B

In example mode B, the occupant detector 110 is trained to emphasizeproviding some violators a free pass at the expense of significantlyreducing the number of honest users that are wrongfully ticketed.Example mode B may be advantageously deployed in a high-occupancy lanewhere it is expected that relatively more high-occupancy vehicles arepresent, such as HOV lanes, to increase faith in the system 100. Inexample mode B, the FP is reduced relative to the FN. Statedalternatively, the FP is reduced at the expense of increasing FN.

TABLE 3 Mode-B-Confusion Matrix where false positives (honest road usersbeing ticketed) is given more importance than false negatives (giving aviolator a pass) Predicted Occupancy 1 occupant 2+ occupants Actual 1occupant 837 163 occupancy 2+ occupants 18 982

Table 3 shows example experimental results where the system 100 istrained to operated according to model B, where false positives (honestroad users being ticketed) is given more importance than false negatives(giving a violator a pass). As is shown in Table 3, this mode makes LESSmistakes on actual 2+ occupant vehicles, and there is a chance of givinga wrong ticket.

According to example embodiments, where the system 100 is setup in a HOVlane and the expectation will be that the majority of road users arehigh-occupancy, the overall system accuracy may be adjusted if mode B isemployed.

In example embodiments, the system 100 can be configured to switchbetween example modes. For example, during an initial phase, it may beexpected that the target lane will experience many cases of a singleoccupant within a detected vehicle travelling in the HOV lanes andviolating the law, and therefore mode A may be employed. This initialphase may include occupants of road vehicles “testing out” the HOV lanesor the system 100, and the tolling system described herein may beconfigured to issue warnings to road users during the initial phase.

As understanding and use of the HOV lanes increases, it may be likelythat the distribution of detected vehicles will shift such that thelarge majority of users are honest high occupancy vehicles. At thattime, the system 100 may be configured to operate according to mode Band experience a system with overall high accuracy. The system 100 usesa method to adaptively change the optimal trade-off of the occupancyovercounting and undercounting errors based on traffic patterns and roaduser behavior. The system learns and adapts overtime from toll orenforcement data gathered and fed back into the system 100 overconsecutive time intervals of the system's 100 operation on the road.”

3-Stage Model

System 100 may be configured to detect the number of occupants in thevehicle, irrespective of a legal requirement for occupancy within aparticular lane. In some embodiments, for example, the calculation forthe vehicle occupancy accuracy may be more complex if multiple optionsare to be determined, such as distinguishing between 1, 2, 3+ occupantswhere are all equally important. The system 100 may be able to achievethe following example performance shown in Table 4:

TABLE 4 Confusion Matrix when all 1, 2, 3+ errors are deemed equallyimportant 1 2 3+ Actual/Predicted occupant occupants occupants 1occupant 90.1%  9.2% 0.7% 2 occupants  9.5% 85.7% 4.8% 3+ occupants   0%  10%  90%

Each of the different modes may be learned during a training stage foreach system 100. The training stage can configure different operatingparameters that corresponds to the desired weighing of FP and FN,balancing the different types of errors and minimizing manualintervention. The system 100 can be adapted to a vast range of FP and FNconditions even after deployment.

The embodiments of the devices, systems and methods described herein maybe implemented in a combination of both hardware and software. Theseembodiments may be implemented on programmable computers, each computerincluding at least one processor, a data storage system (includingvolatile memory or non-volatile memory or other data storage elements ora combination thereof), and at least one communication interface.

FIG. 16 shows an example system with road-side units (iRSU) 1600transmitting data to a cloud server 1602. The iRSU 1600 has an imagingdevice that captures successive images of vehicles moving along in alane on a highway or road.

The systems described throughout the description can be packaged andminiaturized into road side units (iRSU) 1600 with adjustable mechanicaland software components. Multiple such units 1600 can be deployed on aregion of road (highway or urban streets) with different mechanical andsoftware configurations. As a vehicle travels through the region of theroad, the vehicle gets observed by these different units 1600. Each unit1600 can have different mechanical and software configuration, andcaptures its own successive images from different angles andconfigurations and makes its own occupancy prediction. Each unit 1600uploads its unique prediction and the confidence the unit has over theprediction with a unique vehicle identifier (for example: license plate)on a cloud server 1602. Software on the cloud server 1602 then fuses alldata coming from different units 1600 for a single vehicle. This methodproduces a unified higher-fidelity determination for each vehicle whichthen can enable making high accuracy toll or enforcement decisions. Thesystem can be a server 1602 connected to a network of road side units1600 capable of higher level performance (e.g., swarm intelligence).This higher level performance can be obtained because different units1600 can have different configurations and can collectively make betterdecisions based on different fusion methods including but not limited toBayesian estimation and different voting schemes.

The road-side unit 1600 is configured to capture images of vehiclestravelling along a lane of a road. The road side unit can capture imagesfrom a fixed perspective. The successive images are, for each vehicle,analyzed to determine a likely vehicle occupancy. As a result ofcapturing multiple images from the fixed perspective, and further as aresult of the images being captured from a roadside position, the imagesbetween installations may allow for more robust training, and portableoccupancy detection approaches, which are adaptable to a variety ofoperating environments. The use of the multiple images being capturedfrom a fixed roadside position also allows the system to generate arobust estimation of the vehicle occupancy without the need forexpensive or overhead systems that are difficult to install. Theroadside system may require fewer parts, have lower maintenance costs,and be easier to deploy. A central server 1602 can connect to all units1600 and collect data from the units 1600 for processing.

In some embodiments, there is provided a roadside occupancy detector fordetecting vehicle occupancy of a vehicle travelling in an expecteddirection of travel along a road. The roadside occupancy detector (orunit 1600) can have a first roadside imaging device positioned on aroadside, having a first field of view of the road, the first field ofview incident on a side of the vehicle when the vehicle is on the roadwithin the first field of view. The unit 1600 has a first roadside lightemitter emitting light towards vehicles in the first field of view. Theunit 1600 has a roadside vehicle detector. In some embodiments, the unit1600 has a processor, in communication with a memory, configured to:receive a signal from the roadside vehicle detector indicating that thevehicle is within or proximate, relative to the expected direction ofvehicle travel, to the first field of view; command the first roadsidelight emitter to emit light according to a first pattern for a firstduration; command the first roadside imaging device to capture images ofthe side of the vehicle according to a second pattern associated withthe first pattern, during a second duration associated with the firstduration; receive the captured images of the side of the vehicle fromthe first roadside imaging device; compute a vehicle occupancy of thevehicle by, in each of the captured images: determining one or moreregions of interest of the vehicle in each of the captured images;determining the vehicle occupancy as a number of visible occupants inthe one or more regions of interest; and determining a most likelynumber of occupants based on each determined vehicle occupancy; andtransmit the vehicle occupancy to a monitoring system.

FIG. 17 shows another example system with road side units 1600transmitting data to a cloud server 1602. A set of units 1600 canconnect to an intermediate server which can connect to the cloud server1602, in some embodiments. The road side units 1600 can send data to thecloud server 1602 such as occupancy count, confidence value, enforcementdecisions for observed vehicles.

Program code is applied to input data to perform the functions describedherein and to generate output information. The output information isapplied to one or more output devices. In some embodiments, thecommunication interface may be a network communication interface. Inembodiments in which elements may be combined, the communicationinterface may be a software communication interface such as those forinter-process communication. In other embodiments, there may be acombination of communication interfaces implemented as hardware,software, and combination thereof.

In an aspect, a system for detecting occupancy of a vehicle in a road isdisclosed. The system includes an imaging device adjacent to the road,having a field of view of the road. A side of the vehicle is in thefield of view when the vehicle is on the road within the field of view.The system includes a first light emitter adjacent to the road andemitting light towards vehicles in the field of view. There can be avehicle detector adjacent to the road and upstream of the imaging devicerelative to an expected direction of vehicle travel, for example. Aprocessor, in communication with a memory, is configured to receive asignal from the vehicle detector, indicating that the vehicle is withinor upstream, relative to the expected direction of vehicle travel, ofthe field of view. The processor transmits, in response to receiving thesignal, a first command signal to the light emitter to emit lightaccording to a first pattern for a first duration, and, a second commandsignal to the imaging device to capture images of the side of thevehicle according to a second pattern associated with the first pattern,for a second duration associated with the first duration. The processorreceives the captured images of the side of the vehicle from the imagingdevice, and computes a vehicle occupancy of the vehicle based on thecaptured images. The processor transmits the vehicle occupancy to amonitoring system.

In example embodiments, the imaging device, the first light emitter, andthe vehicle detector are attached to a mobile roadside structureadjacent to the road at a height taller than an expected barrier.

In example embodiments, the imaging device is positioned to have a yawangle relative to a horizontal axis perpendicular to the expecteddirection of vehicle travel such that each of the images captured by theimaging device includes different perspectives of the vehicle based onthe first yaw angle. In example embodiments, the different perspectivesare empirically determined so as to yield different angles of anyoccupants in a rear and front side window.

In some embodiments, the imaging device can capture multiple images fromthe vehicle so that occupants are seen from different perspectives asthe vehicle travels horizontally across the field of view, which can bereflected in extracted data from the images. An imaging device can havea large horizontal field of view and the system may be able to extractdata for change of perspective by taking multiple successive images asthe vehicle is traveling from one end of the horizontal field of view tothe other, even with a zero yaw angle. However, having a large of afield of view may not always possible or favorable for some otherreasons. Using a nonzero yaw angle may accentuate the change ofperspective within a limited horizontal motion of the vehicle in thefield of view. Accordingly, the system computes data corresponding tochange of perspective, and, in some embodiments uses the yaw angle as anexample.

In example embodiments, the system further includes a second imagingdevice, adjacent to the road, a first height above the ground greaterthan a height above the ground of the imaging device. The second imagingdevice has a field of view of a second lane of the road, the second lanebeing further from the first imaging device than the first lane of theroad. A side of a further vehicle is in the second field of view whenthe further vehicle is in the second lane within the second field ofview. The system includes a second light emitter adjacent to the roadand emitting light towards vehicles in the field of view of the secondlane. The processor is further configured to receive a second signalfrom the vehicle detector indicating that a further vehicle is within orupstream, relative to the expected direction of vehicle travel, of thefield of view of the second lane. The processor transmits, in responseto receiving the second signal, a third command signal to the secondlight emitter to emit light according to a third pattern for a thirdduration, and a fourth command signal to the second imaging device tocapture additional images of a side of the further vehicle according toa fourth pattern associated with the third pattern, for a fourthduration associated with the third duration. The processor receives theadditional captured images of the side of the further vehicle from thesecond imaging device, and computes a vehicle occupancy of the furthervehicle based on the additional captured images. The processor transmitsthe vehicle occupancy of the further vehicle to the monitoring system.

In example embodiments, to compute the vehicle occupancy of the vehicle,the processor is further configured to, in each of the captured images:determine one or more regions of interest of the vehicle in each of theplurality of captured images; reduce the plurality of captured images tothe determined one or more regions of interest; determine a number ofvisible occupants in the reduced plurality of images; and determine amost likely number of occupants based on an occupant model most likelyto fit all of the respective number of visible occupants in the capturedimages.

In example embodiments, the processor is further configured to monitorsignals over time to determine an expected vehicle speed of the vehicleand adjust one or more parameters of the imaging device into adetermined optimal configuration for capturing vehicles travelling theexpected vehicle speed.

In example embodiments, the processor is further configured to monitorsignals over time to determine an expected speed of the vehicle, anddetermine the first pattern and the first time window based on theexpected vehicle speed.

In example embodiments, the processor is further configured to monitorsignals over time to determine an expected speed of the vehicle, anddetermine one or more normalization parameters, the one or morenormalization parameters adjusting the representation of the vehicle inthe images to account for the expected vehicle speed. The processorgenerates a normalized vehicle model populated with each of theplurality of images processed with the respective normalizationparameters to normalize the vehicle across the captured plurality toimages.

In example embodiments, the system further includes a sensor fordetecting ambient conditions, and the processor is further configured toreceive ambient condition information from the sensor, determine anoptimal configuration for the imaging device based on the receivedambient condition, and transmit a further command signal to the imagingdevice capture images according to the optimal configuration. In exampleembodiments, the optimal configuration is an imaging device exposuregain or aperture.

In example embodiments, the light emitter is an LED emitting infrared ornear infrared light and the first pattern is 120 pulses per second.

In example embodiments, the system further includes a license plateimaging device, and the processor is further configured to transmitanother command signal to the license plate imaging device to captureimages of the rear of the vehicle, and compute a license plate based onthe rear end captured image.

In a further aspect, a method of configuring a system for detectingoccupancy of a vehicle proximate to a road is disclosed. The methodincludes determining a distance from a target roadside mounting locationto a target lane of the road, determining a width of the target lane,and determining a difference between the height of the ground at thetarget roadside mounting location and a height of the location of thetarget lane. The method includes determining a preferred mountinggeometry of a light emitter, an imaging device, and a vehicle detectorat the target roadside mounting location based on the width, thedistance, and the difference; and installing the light emitter, theimaging device, and the vehicle detector to a mobile platform at thetarget roadside mounting location to enable the imaging device tocapture successive images of a detected vehicle.

The method, in example embodiments, includes monitoring mountedpositions of the light emitter, the imaging device, and the vehicledetector to determine whether the mounted positions coincide with thepreferred mounting geometry on a display, and in response to determiningthe mounted positions of the light emitter, the imaging device, and thevehicle detector do not coincide with the preferred mounting geometry,displaying an error message on the display.

In example embodiments, the error message includes a correctionparameter. Accordingly, the method can involve computing a correctionparameter and providing visual guidance using augmented reality avatarson a display device. For example the visual guidance can involve showingsemi-transparent virtual lane markings overlaid on camera feed andasking the calibration technician to move the camera until the virtuallane markings match and cover the real lane markings on the road in theimage feed displayed on the screen.

Throughout the following discussion, numerous references will be maderegarding servers, services, interfaces, portals, platforms, or othersystems formed from computing devices. It should be appreciated that theuse of such terms is deemed to represent one or more computing deviceshaving at least one processor configured to execute softwareinstructions stored on a computer readable tangible, non-transitorymedium. For example, a server can include one or more computersoperating as a web server, database server, or other type of computerserver in a manner to fulfill described roles, responsibilities, orfunctions.

The following discussion provides many example embodiments. Althougheach embodiment represents a single combination of inventive elements,other examples may include all possible combinations of the disclosedelements. Thus if one embodiment comprises elements A, B, and C, and asecond embodiment comprises elements B and D, other remainingcombinations of A, B, C, or D, may also be used.

The term “connected” or “coupled to” may include both direct coupling(in which two elements that are coupled to each other contact eachother) and indirect coupling (in which at least one additional elementis located between the two elements).

The technical solution of embodiments may be in the form of a softwareproduct. The software product may be stored in a non-volatile ornon-transitory storage medium, which can be a compact disk read-onlymemory (CD-ROM), a USB flash disk, or a removable hard disk. Thesoftware product includes a number of instructions that enable acomputer device (personal computer, server, or network device) toexecute the methods provided by the embodiments.

The embodiments described herein are implemented by physical computerhardware, including computing devices, servers, receivers, transmitters,processors, memory, displays, and networks. The embodiments describedherein provide useful physical machines and particularly configuredcomputer hardware arrangements. The embodiments described herein aredirected to electronic machines and methods implemented by electronicmachines adapted for processing and transforming electromagnetic signalswhich represent various types of information. The embodiments describedherein pervasively and integrally relate to machines, and their uses;and the embodiments described herein have no meaning or practicalapplicability outside their use with computer hardware, machines, andvarious hardware components. Substituting the physical hardwareparticularly configured to implement various acts for non-physicalhardware, using mental steps for example, may substantially affect theway the embodiments work. Such computer hardware limitations are clearlyessential elements of the embodiments described herein, and they cannotbe omitted or substituted for mental means without having a materialeffect on the operation and structure of the embodiments describedherein. The computer hardware is essential to implement the variousembodiments described herein and is not merely used to perform stepsexpeditiously and in an efficient manner.

Although the embodiments have been described in detail, it should beunderstood that various changes, substitutions and alterations can bemade herein without departing from the scope as defined by the appendedclaims.

Moreover, the scope of the present application is not intended to belimited to the particular embodiments of the process, machine,manufacture, and composition of matter, means, methods and stepsdescribed in the specification. As one of ordinary skill in the art willreadily appreciate from the disclosure of the present invention,processes, machines, manufacture, compositions of matter, means,methods, or steps, presently existing or later to be developed, thatperform substantially the same function or achieve substantially thesame result as the corresponding embodiments described herein may beutilized. Accordingly, the appended claims are intended to includewithin their scope such processes, machines, manufacture, compositionsof matter, means, methods, or steps

As can be understood, the examples described above and illustrated areintended to be exemplary only.

1. A system for detecting vehicle occupancy, the system comprising: afirst roadside imaging device having a first field of view; a firstroadside light emitter emitting light in the first field of view; aroadside vehicle detector; a processor, in communication with a memory,configured to: receive a signal from the roadside vehicle detector;command the first roadside light emitter to emit light according to afirst pattern for a first duration; command the first roadside imagingdevice to capture one or more images according to a second patternassociated with the first pattern, during a second duration associatedwith the first duration; receive the captured images from the firstroadside imaging device; compute a vehicle occupancy by, in each of thecaptured images: determining one or more regions of interest in each ofthe captured images; determining the vehicle occupancy based on the oneor more regions of interest; and determining a most likely number ofoccupants based on each determined vehicle occupancy; and transmit thevehicle occupancy to a monitoring system or store the vehicle occupancyin memory.
 2. The system of claim 1, wherein: the first roadside imagingdevice is positioned to extract data for different perspectives acrossthe field of view; and at least some of the images captured by the firstroadside imaging device include the different perspectives.
 3. Thesystem of claim 2 wherein the processor is configured to compute a yawangle relative to a horizontal axis perpendicular to an expecteddirection, wherein the images captured by the first roadside imagingdevice include the different perspectives based on the first yaw angle.4. The system of claim 1, wherein the processor, to compute the vehicleoccupancy, is configured to: discard uninteresting regions of theplurality of captured images to generate subsets of the plurality ofcaptured images; and determine a number of visible occupants based ondetermining one or more regions of interest in the respective subset ofthe plurality of captures images.
 5. The system of claim 1, wherein thefirst roadside imaging device, the first roadside light emitter, and thevehicle detector are attached to a mobile roadside structure.
 6. Thesystem of claim 1, further comprising: a second roadside imaging device,above the first roadside imaging device, the second roadside imagingdevice having a second field of view; a second roadside light emitteremitting light in the second field of view; wherein the processor isfurther configured to: receive another signal from the vehicle detector;command the second roadside light emitter to emit light according to athird pattern for a third duration; command the second roadside imagingdevice to capture additional images according to a fourth patternassociated with the third pattern, during a fourth duration associatedwith the third duration; receive the additional captured images from thesecond roadside imaging device; compute another vehicle occupancy by, ineach of the additional captured images by: determining one or moreregions of interest in each of the additional captured images;determining the vehicle occupancy using the one or more regions ofinterest; and determining a most likely number of occupants based oneach determined vehicle occupancy of the further vehicle; and transmitthe vehicle occupancy to the monitoring system.
 7. The system of claim6, wherein the first field of view and the second field of view overlap,and the processor is further configured to: determine the one or moreregions of interest in the one or more additional captured images;determine a further number of visible occupants in the one or moreadditional captured images in the one or more regions of interest; anddetermine the most likely number of occupants based on each determinedvehicle occupancy and each determined further number of visibleoccupants.
 8. The system of claim 1, wherein the processor is furtherconfigured to: monitor, over time, a plurality of signals from theroadside vehicle detector to determine an expected vehicle speed; andadjust one or more parameters of the first roadside imaging device orthe first light emitter into a determined optimal configuration forcapturing images based on the expected vehicle speed.
 9. The system ofclaim 1, wherein the processor is further configured to: monitor, overtime, a plurality of signals from the roadside vehicle detector todetermine an expected vehicle speed; and determine the first pattern andthe first time window based on the expected vehicle speed.
 10. Thesystem of claim 1, further comprising: a sensor for detecting ambientconditions; wherein the processor is further configured to: receiveambient condition information from the sensor; determine an optimalconfiguration for the imaging device based on the received ambientcondition; and transmit a further command signal to the imaging devicecapture images according to the optimal configuration.
 11. The system ofclaim 1, wherein the light emitter is an LED emitting infrared or nearinfrared light, the first pattern is 120 pulses per second.
 12. A methodfor detecting vehicle occupancy, the method comprising; receiving asignal from a detector based on a first field of view of a firstroadside imaging device; commanding a first roadside light emitter toemit light according to a first pattern for a first duration; commandingthe first roadside imaging device to capture images according to asecond pattern associated with the first pattern, during a secondduration associated with the first duration; receiving the capturedimages from the first roadside imaging device; computing a vehicleoccupancy by, in each of the captured images: determining one or moreregions of interest in each of the captured images; determining thevehicle occupancy in the one or more regions of interest; anddetermining a most likely number of occupants based on each determinedvehicle occupancy; and transmitting the most likely number of occupantsto a monitoring system or storing the vehicle occupancy in memory. 13.The method of claim 12, further comprising: discarding uninterestingregions of the plurality of captured images to generate subsets of theplurality of captured images; and determining the number of occupantsbased on determining one or more regions of interest in the respectivesubset of the plurality of captures images.
 14. The method of claim 12,wherein the one or more regions of interest include at least one of arear side window and a front side window.
 15. The method of claim 12,wherein each of the captured images includes different perspectivesbased on a yaw angle which encourages image variation.
 16. The method ofclaim 12, the method further comprising: commanding a second roadsideimaging device to capture additional images from a second field of viewaccording to a fourth pattern associated with the first pattern, for afourth duration associated with the first duration; receive theadditional captured images from the second roadside imaging device;wherein computing the vehicle occupancy further comprises, for each ofthe additional captured images: determining one or more additionalregions of interest of the vehicle; determining the vehicle occupancy inthe additional one or more regions of interest; and determining the mostlikely number of occupants based on the each of the number of visibleoccupants and the further number of visible occupants; and transmittingthe vehicle occupancy to the monitoring system.
 17. The method of claim16, the method further comprising: receiving a signal indicating fromthe detector based on the second field of view; commanding a secondroadside light emitter to emit light according to a third pattern for athird duration; commanding the second roadside imaging device to captureadditional images according to a fourth pattern associated with thethird pattern, during a fourth duration associated with the thirdduration; receiving the additional captured images from the firstroadside imaging device; computing a further vehicle occupancy by, ineach of the additional captured images: determining one or more furtherregions of interest in each of the additional captured images;determining the further vehicle occupancy based on the one or morefurther regions of interest; and determining a most likely number ofoccupants based on each determined further vehicle occupancy; andtransmitting the most likely number of occupants to the monitoringsystem.
 18. The method of claim 11 further comprising computing acorrection parameter and providing visual guidance using augmentedreality avatars on a display device.
 19. The method of claim 12, themethod further comprising: monitoring, over time, a plurality of signalsfrom the roadside vehicle detector to determine an expected vehiclespeed; and adjusting one or more parameters of the first roadsideimaging device or the first light emitter into a determined adjustedconfiguration for capturing images based on the expected vehicle speed.20. The method of claim 12, the method further comprising: monitoring,over time, a plurality of signals from the roadside vehicle detector todetermine an expected vehicle speed; and determining the first patternand the first time window based on the expected vehicle speed.